Endpoint integration

Mount createEvalCanaryEndpoint as the POST handler for a dedicated server route. One endpoint is bound to one project and can dispatch several agent keys.

import { conversationAgent, createEvalCanaryEndpoint } from "evalcanary/target";

const projectId = process.env.EVALCANARY_PROJECT_ID;
const webhookSecret = process.env.EVALCANARY_WEBHOOK_SECRET;
if (!projectId) throw new Error("EVALCANARY_PROJECT_ID is missing");
if (!webhookSecret) {
  throw new Error("EVALCANARY_WEBHOOK_SECRET is missing");
}

export const POST = createEvalCanaryEndpoint({
  webhookSecret,
  projectId,
  revision: process.env.APP_REVISION,
  agents: {
    "support-chat": conversationAgent({
      execute: ({ messages, signal }) => runSupportAgent({ messages, signal }),
    }),
  },
});

Pass the original Fetch Request directly to this handler. Do not parse, read, or reserialize the body first; the SDK authenticates the exact bytes sent by EvalCanary before it parses the request.

conversationAgent validates a conversation ending in a non-empty user message. Its execute function can return a string or { output, evidence?, llmCalls? }. Keep the called function shared with your customer-facing path so the check uses the same application behavior.

Set revision to an immutable build identifier when one is available. EvalCanary records it with every sample.

Connect the route

Deploy the route, then save its URL under Project settings → Production endpoint. The URL must be publicly reachable HTTPS without credentials, a query string, or a fragment. It must accept the request directly; redirects and deployment login pages are not supported.

Create an endpoint signing secret in Project settings and store it as EVALCANARY_WEBHOOK_SECRET. It is shown in plaintext only once. The SDK verifies requests locally, so the endpoint does not need to call EvalCanary before running your agent.

Under Agent types, register every exposed agent with the exact key and matching request format. Agent and actor keys use lowercase identifiers such as support-chat and vip-customer.

AI SDK adapter

Use aiSdkConversationAgent when the application returns an AI SDK 7 generateText or streamText result:

import { aiSdkConversationAgent } from "evalcanary/ai-sdk";

const supportAgent = aiSdkConversationAgent({
  execute: ({ messages, signal }) => runSupportAgent({ messages, signal }),
});

The adapter waits for completed steps and returns the final text, normalized tool calls, and one LLM usage entry per completed model step. Each entry can contain provider, model, provider request ID, and input, output, cache, reasoning, and total token counts. EvalCanary calculates an estimated cost when pricing and token accounting are known; the endpoint does not send prices.

Typed agents, actors, and context

Use a contract when the route exposes multiple input shapes or a scenario needs a synthetic identity. actorAliases is a finite allowlist. resolveContext maps an alias to application-owned context before the selected agent executes.

import { aiSdkConversationAgent } from "evalcanary/ai-sdk";
import type {
  EvalCanaryConversationAgentDefinition,
  EvalCanaryJsonAgentDefinition,
} from "evalcanary/target";
import { createEvalCanaryEndpoint, jsonAgent } from "evalcanary/target";
import { z } from "zod";

const classifierInput = z.strictObject({ text: z.string().min(1) });
type ClassifierInput = z.infer<typeof classifierInput>;
type AppContext = { downstreamHeaders: Headers };

type Contract = {
  actors: "standard-customer" | "vip-customer";
  agents: {
    "support-chat": EvalCanaryConversationAgentDefinition<AppContext>;
    classifier: EvalCanaryJsonAgentDefinition<ClassifierInput, AppContext>;
  };
};

const projectId = process.env.EVALCANARY_PROJECT_ID;
const webhookSecret = process.env.EVALCANARY_WEBHOOK_SECRET;
if (!projectId) throw new Error("EVALCANARY_PROJECT_ID is missing");
if (!webhookSecret) {
  throw new Error("EVALCANARY_WEBHOOK_SECRET is missing");
}

export const POST = createEvalCanaryEndpoint<Contract>({
  projectId,
  webhookSecret,
  revision: process.env.APP_REVISION,
  actorAliases: {
    "standard-customer": true,
    "vip-customer": true,
  },
  async resolveContext({ actor }) {
    if (!actor) throw new TypeError("A test profile is required");
    return {
      downstreamHeaders: await headersForSyntheticActor(actor),
    };
  },
  agents: {
    "support-chat": aiSdkConversationAgent<AppContext>({
      execute: ({ context, messages, signal }) =>
        runSupportAgent({
          headers: context.downstreamHeaders,
          messages,
          signal,
        }),
    }),
    classifier: jsonAgent<ClassifierInput, AppContext>({
      inputSchema: classifierInput,
      execute: ({ context, input, signal }) =>
        runClassifier({
          headers: context.downstreamHeaders,
          signal,
          text: input.text,
        }),
    }),
  },
});

Create matching entries under Test profiles. EvalCanary sends only the selected profile key. Cookies, tokens, and service headers created by resolveContext remain inside your application unless your code returns them.

resolveContext also receives the typed agent, typed request, and request abort signal, so context can vary by agent without accepting arbitrary identity values.

Request and response

EvalCanary sends one request for each scenario sample:

type EvalCanaryRequest = {
  schemaVersion: "1";
  requestId: string;
  projectId: string;
  checkId: string;
  scenarioId: string;
  runId: string;
  sample: number;
  agent: string;
  actor?: string;
  input: JsonValue;
};

The SDK returns:

type EvalCanaryResponse = {
  schemaVersion: "1";
  output: JsonValue;
  evidence?: EvalCanaryEvidenceV1;
  extensions?: Record<string, JsonValue>;
  revision?: string;
  llmCalls?: EvalCanaryLlmCall[];
};

Use output for the response being checked. Use typed evidence for bounded activity such as { toolCalls: [{ name, status: "succeeded", input, output }] }. Each tool call must report status as requested, succeeded, or failed; include output for succeeded calls and an error for failed calls. Tool-call pass conditions read tool names from that shape. All values must be plain JSON; Date, BigInt, class instances, and non-finite numbers are rejected.

JsonValue is the compile-time TypeScript JSON shape. Runtime validation also applies the SDK's wire bounds for depth, collection sizes, strings, and bytes. The AI SDK adapter automatically records only tool id, name, and status (with a fixed generic failure message when required); it does not capture tool inputs, outputs, or raw error details. Additional evidence should be explicitly and manually sanitized.

extensions is a bounded namespaced record for application-specific response metadata that EvalCanary does not interpret. Endpoint errors use the same schemaVersion: "1" envelope and one of the SDK's exported error codes.

Requests and responses are limited to 64 KiB by default. The SDK rejects unknown agents, actors, fields, and invalid JSON input before application code runs.