Runs and evidence
Understand run execution, sample evidence, and LLM usage.
One run executes one saved check version. It can start from Run now or from the check's configured schedule. The check must be enabled and the project must have both a production endpoint and an endpoint signing secret before either trigger can start it.
When a run is queued, EvalCanary fixes the check version, endpoint URL, agent ID, and scenario test-profile IDs. It resolves the current agent and test-profile keys when execution begins, so changing or deleting them before then affects that queued run.
Runs originate from EvalCanary's hosted service, so the endpoint must be publicly reachable. Local development endpoints remain available for conversation preview but cannot run checks until they are deployed or exposed through a public HTTPS tunnel.
EvalCanary sends one authenticated POST request for every sample. For example, two scenarios with three samples each produce six endpoint calls. There is no additional process to operate in your environment.
Sample evidence
Each completed sample can include:
- scenario ID, sample number, timestamps, and outcome;
- scenario input and the complete endpoint request;
- raw endpoint response,
output, optional typedevidence, and target revision; - every pass condition's outcome, message, and structured details; and
- bounded execution error details when the endpoint did not complete.
The run page keeps this structure nested by sample so a response can be traced back to the exact scenario, request, and check version that evaluated it.
Run outcomes
| Outcome | Meaning |
|---|---|
| Passed | Every scenario produced enough determinate samples and met the required pass rate. |
| Failed | At least one scenario's determinate pass rate was below the threshold. |
| Indeterminate | At least one scenario did not produce its minimum number of determinate samples. |
| Error | Every sample ended in an execution or evaluation error. |
A known pass-condition failure remains failed even when another pass condition errors. Open an indeterminate or errored sample to distinguish missing tool evidence, evaluator limits, authentication failures, endpoint failures, and timeouts.
Reported LLM usage
The endpoint can report an llmCalls array with provider, model, provider request ID, and token counts. aiSdkConversationAgent derives one entry from each completed AI SDK step. Provider-neutral adapters can return the same fields directly. Provider retries and calls that fail before returning usage are not available through this interface, so these numbers are reported usage rather than a provider invoice.
The run page groups usage by purpose, provider, and model:
- Agent response usage came from the application endpoint.
- AI evaluation usage came from AI judgment evaluation.
EvalCanary calculates an estimated cost once from a versioned pricing catalog when the provider, model, and token accounting are known and internally consistent. It stores the final estimate and catalog version, so a later catalog update cannot rewrite history. Calls with unknown pricing or inconsistent usage remain visible with cost shown as unavailable. Customer endpoints report usage, not prices.
Retention
Detailed sample evidence follows the project's retention policy. When it expires, the run record and LLM usage records remain available, while sample counts and details are no longer shown. Keep sensitive identity context out of scenario input, output, evidence, and tool data; see Auth and privacy.
Failure and recovery deliveries are configured separately in Webhook notifications.