Checks and pass conditions

A check sends one or more scenarios to a registered agent. Pass conditions decide whether each response succeeds. The API calls pass conditions assertions.

Editing a check creates a new immutable version for future runs. Existing runs remain linked to the exact version they used.

Every pass condition applies to every scenario in the check. Create another check when two scenarios need different expectations.

Pass condition types

Keep each pass condition focused on one result. A refund check, for example, can require an order lookup, require an escalation, and forbid issuing the refund as three separate pass conditions.

Tool-call pass conditions verify that a call was attempted. They do not prove that the tool completed successfully. Tool results, arguments, ordering, and call counts are not supported yet.

Reporting tool activity

Tool-call pass conditions require endpoint evidence shaped like this:

{
  "toolCalls": [
    {
      "name": "lookup_order",
      "status": "succeeded",
      "input": { "orderId": "EC-1042" },
      "output": { "found": true }
    }
  ]
}

Use requested when a call was started but no result is available, succeeded when it completed with an output, and failed when it completed with an error. Report { "toolCalls": [] } when the endpoint captured tool activity and no tools were called. Omit toolCalls only when the endpoint cannot determine what happened. Missing or malformed tool evidence makes tool-call pass conditions Indeterminate.

The AI SDK adapter reports this evidence automatically, including an empty array when no tools were called.

Sampling thresholds

Samples per scenario is the number of times each scenario runs. A run contains:

scenario count × samples per scenario

A definition can contain up to 10 scenarios and 10 pass conditions, with at most 10 total samples per run. Minimum valid results per scenario and Required pass rate per scenario are applied independently to every scenario. One healthy scenario cannot hide failures in another. The pass rate must be between 1% and 100%.

Start with one scenario, one call, one minimum valid result, and a 100% pass rate. Increase the call count only when the behavior is variable enough to need sampling.

Outcomes

Every pass condition produces Pass, Fail, Indeterminate, or Error. A known failure remains a failure even if another pass condition cannot complete. A sample uses this order: Fail, Error, Indeterminate, then Pass.

The run result is derived as follows:

1. If every sample errored, the run is Error.
2. If any scenario's determinate pass rate is below the configured threshold, the run is Fail.
3. If no scenario failed but any scenario has fewer determinate samples than its minimum, the run is Indeterminate.
4. Otherwise, the run is Pass.

Errored and indeterminate samples stay visible in the run, but they are not counted as passed or failed in the final rate.