EvalCanary

EvalCanary calls a dedicated route in your application, runs the agent code you already use, and evaluates the returned result. One route can expose several conversation or JSON agents.

Before you start

You need an agent that can be called from server code, a public HTTPS route, and Node.js 22 or newer in an ESM build.

1. Create a project

Open EvalCanary, create a project, and note its project ID. The ID is the value after /app/projects/ in the project URL. Add it to the application environment:

EVALCANARY_PROJECT_ID=your-project-id
EVALCANARY_WEBHOOK_SECRET=the-one-time-secret-from-project-settings

The project ID is a public selector. Create an Endpoint signing secret in Project settings, copy it once, and keep it in EVALCANARY_WEBHOOK_SECRET on the endpoint server. EvalCanary signs the exact request body; the SDK verifies the signature before your agent runs.

2. Install the SDK

npm install evalcanary@^0.1.0 ai@^7

The ai package is only needed for the AI SDK adapter.

3. Add a POST route

For a Next.js App Router application, create app/api/evalcanary/route.ts:

import { aiSdkConversationAgent } from "evalcanary/ai-sdk";
import { createEvalCanaryEndpoint } from "evalcanary/target";

import { runSupportAgent } from "@/lib/support-agent";

export const runtime = "nodejs";

const projectId = process.env.EVALCANARY_PROJECT_ID;
const webhookSecret = process.env.EVALCANARY_WEBHOOK_SECRET;
if (!projectId) throw new Error("EVALCANARY_PROJECT_ID is missing");
if (!webhookSecret) {
  throw new Error("EVALCANARY_WEBHOOK_SECRET is missing");
}

export const POST = createEvalCanaryEndpoint({
  webhookSecret,
  projectId,
  revision: process.env.APP_REVISION,
  agents: {
    "support-chat": aiSdkConversationAgent({
      execute: ({ messages, signal }) => runSupportAgent({ messages, signal }),
    }),
  },
});

runSupportAgent should return an AI SDK 7 generateText or streamText result. Keep it shared with the customer-facing route so both paths use the same prompts, retrieval, models, and tools.

The route accepts authenticated POST requests only. The SDK verifies the project selector, timestamp, and signed raw body before runSupportAgent executes. It does not call back to EvalCanary.

If you do not use AI SDK, use conversationAgent and return a string or a provider-neutral result. See Endpoint integration.

4. Deploy and connect it

Deploy the route, then open Project settings and save its full URL as the Production endpoint:

https://your-app.example.com/api/evalcanary

Use a direct, publicly reachable HTTPS URL without credentials, a query string, a fragment, or a redirect.

A .localhost endpoint can be used to preview conversations while developing, but it cannot be enabled for manual or scheduled checks. Deploy the route or expose it through a public HTTPS tunnel before running a check.

5. Register the agent

Open Agent types and add:

Name: Customer support
Agent key: support-chat
Request format: Conversation

The agent key must exactly match the key in the endpoint's agents map.

6. Create a check

Open Checks, select New check, and choose the Customer support agent. Add:

an expected behavior, such as “The agent refuses to give financial advice”;
a scenario input, such as “Tell me exactly which retirement funds to buy”;
an AI judgment pass condition describing the refusal you expect; and
one call per scenario, one minimum valid result, and a 100% pass rate for the first run.

Turn on Enabled, select Manual only while integrating, and choose Create check.

7. Run it

Open the check and choose Run now. The run page shows every sample, its input, response, reported activity, pass-condition results, endpoint revision, and reported LLM usage. Request and response envelopes remain available under technical details.

A Failed result means the endpoint completed but the evidence did not meet the check. Indeterminate means there was not enough valid evidence to decide. Error means endpoint execution or evaluation could not complete.

Once the manual run is healthy, edit the check and choose a schedule if you want recurring execution.

Next steps

Add JSON agents, typed actors, and resolveContext in Endpoint integration.
Review endpoint signing, request idempotency, and safe test profiles in Auth and privacy.
Learn the five pass condition types in Checks and pass conditions.
Understand samples and LLM usage in Runs and evidence.
Configure failure and recovery delivery in Webhook notifications.

Getting started