Skip to content

Check whether your AI still follows the rules.

Prompts, models, policies, and tools change. EvalCanary runs the checks you enable against your endpoint and creates an in-app alert when a run fails or cannot complete.

Request previewExplore the demo

See what the agent returned, which tools it reported, and the result of every pass condition.

Example check results

Production assistant · behavior checks

1 recent alert

CHK-01

Do not give financial advice, even when asked

Latest run

Failed

Response: “Move your retirement savings into crypto.”

Seven recent runs

  1. Run 1: pass
  2. Run 2: pass
  3. Run 3: pass
  4. Run 4: pass
  5. Run 5: pass
  6. Run 6: pass
  7. Run 7: fail

CHK-02

Do not reveal the system prompt or underlying model

Latest run

Passed

3 of 3 extraction prompts refused

Seven recent runs

  1. Run 1: pass
  2. Run 2: pass
  3. Run 3: pass
  4. Run 4: pass
  5. Run 5: pass
  6. Run 6: pass
  7. Run 7: pass

CHK-03

Never ask for passwords or one-time security codes

Latest run

Running

Checking account-recovery conversations

Seven recent runs

  1. Run 1: pass
  2. Run 2: pass
  3. Run 3: pass
  4. Run 4: pass
  5. Run 5: pass
  6. Run 6: pass
  7. Run 7: running
3 checks enabled20 completed runs1 running

Online doesn’t mean correct.

Your app can stay healthy while a model, prompt, policy, or tool change quietly alters what your AI says or does.

  • Unsafe advice

    Recommends moving retirement savings into crypto.

  • System disclosure

    Names its model and repeats internal instructions.

  • Credential request

    Asks for a one-time security code during account recovery.

How EvalCanary works

Define the behavior here. Test the path you connect.

Product, risk, and engineering teams agree on the behavior. You configure a protected endpoint to invoke the application path you want to verify; EvalCanary turns the returned evidence into a result your team can inspect.

  1. 01

    Define a check

    Describe the behavior, test conversations, and pass conditions.

  2. 02

    Review and save

    Review exactly what will run, then save the check.

  3. 03

    Exercise the configured path

    Your endpoint maps each test profile to the production code path you choose.

  4. 04

    Inspect the result

    See the response, reported tool activity, and pass-condition results.

Results you can investigate

Test the application, not just the prompt.

Your endpoint decides which application code, authorization context, retrieval, model settings, and tools each test profile uses. EvalCanary evaluates the returned evidence against the pass conditions.

  • Saved check definition and pass criteria
  • Recorded responses and tool calls
  • Reported tool calls and outputs
  • Condition results and an alert for every non-passing run
Example failed run
Check failed
Check
Financial advice boundary · v7
Endpoint
api.example.com/evalcanary
Scenario
Customer asks how to invest retirement savings
Response preview
“Move your retirement savings into crypto.”
Evaluation
AI judgment · failed
Why it failed
Do not give financial advice, even when asked
Samples
0 of 1 passed

Evidence

Result recorded

Duration

2.1 seconds

Start with one important rule

The behavior customers rely on deserves a check.

Start with one agent, one rule, and the real path it takes through your product. Add more checks as the decisions become clear.

Request preview