Test Sets & Dynamic Runs

Test Sets let you define reusable collections of test cases and section subscriptions, then create Test Runs from them through the API. They are the recommended way to drive automated run creation from CI/CD pipelines or automation frameworks.

What is a Test Set?#

A Test Set is a project-scoped template that contains two kinds of items:

Item type	Behavior
`case`	A static reference to a specific test case. The case is resolved once at run creation time.
`section`	A dynamic subscription to a section. New cases added to the section later automatically join any `planned` or `in_progress` run that subscribed to it.

Use section items when you want runs to stay in sync with the library as your team adds new cases. Use case items when you need a fixed, repeatable subset.

Required API scopes#

Operation	Scope
Read test sets / preview	`testsets:read`
Create test set	`testsets:create`
Update test set	`testsets:update`
Delete test set	`testsets:delete`
Create test runs from sets	`testruns:create`
Split a run	`testruns:create`

Create a Test Set#

POST /api/v1/projects/{id}/test-sets
Content-Type: application/json
Authorization: Bearer <api_key>

{
  "name": "Sprint Smoke",
  "description": "Smoke coverage for every sprint",
  "items": [
    { "item_type": "case", "case_id": "<case-uuid>", "position": 0 },
    { "item_type": "section", "section_id": "<section-uuid>", "position": 1 }
  ]
}

position controls the order items appear in. A set can contain up to 1,000 items.

Preview a Test Set#

Before creating a run, you can preview what cases a set currently resolves to:

GET /api/v1/projects/{id}/test-sets/{setId}/preview
Authorization: Bearer <api_key>

The response includes the current total case count and a sample of cases:

{
  "id": "<set-id>",
  "name": "Sprint Smoke",
  "current_case_count": 42,
  "sample_cases": [
    { "id": "<case-id>", "title": "Login", "display_code": "TC-MOB-0001" }
  ]
}

Create a Test Run from Test Sets#

POST /api/v1/testruns
Content-Type: application/json
Authorization: Bearer <api_key>

{
  "name": "Sprint 12 Regression",
  "project_id": "<project-uuid>",
  "test_set_ids": ["<set-id-1>", "<set-id-2>"],
  "mode": "unified",
  "environment": "staging",
  "release_id": "<release-uuid>",
  "build_id": "<build-uuid>"
}

mode: "unified" creates a single run containing every resolved case and section subscription.
mode: "split_by_set" creates one run per test set. Each child run is tagged with source_test_set_id.
test_case_ids and test_set_ids are mutually exclusive.

Dynamic section subscriptions are only attached in unified mode. Runs created with split_by_set are independent static runs.

Dynamic section behavior#

When a test case is created or moved into a section that an in-flight run subscribes to, the case automatically becomes an execution in that run. Runs in planned or in_progress status receive new cases; submitted, completed, aborted, and superseded runs do not.

If a case is moved out of a subscribed section:

planned runs: the execution is removed.
in_progress runs: the execution is removed only if it has not yet been executed (result IS NULL).
Executed cases are never removed automatically.

This fan-out is best-effort. The underlying create/move/delete operation succeeds even if subscription propagation fails.

Split a Test Run#

Splitting moves unexecuted cases from an existing run into one or more child runs.

POST /api/v1/testruns/{id}/split
Content-Type: application/json
Authorization: Bearer <api_key>

{
  "strategy": "by_section",
  "name": "Sprint 12 Regression — Split",
  "assignee_id": "<user-uuid>"
}

Strategies:

Strategy	Behavior
`by_section`	One child run per section that has unexecuted cases.
`manual`	Move only the `execution_ids` you provide.
`by_set`	Move all portable cases into a single child run (no set provenance is preserved on executions).

Executed cases stay locked in the original run. If the original run is left with no executions, its status becomes superseded. The original run keeps its dynamic subscriptions; new cases in subscribed sections still land there.

Common errors#

Error	Cause
`VALIDATION_ERROR`	Missing required fields, invalid UUIDs, or mutually exclusive `test_case_ids`/`test_set_ids`.
`RESOURCE_NOT_FOUND`	Project, test set, run, release, or build does not exist or is not accessible.
`FORBIDDEN`	API key scope or project scoping does not allow the operation.
`CONFLICT`	Split is requested on a run that has no portable (unexecuted) cases.