Auth and privacy

Endpoint signing

Each endpoint is bound to one project ID and one endpoint signing secret. Create the secret in Project settings, copy it once, and store it only on the server that hosts your endpoint:

EVALCANARY_WEBHOOK_SECRET=evc_whsec_live_...

Pass it to the endpoint helper:

const webhookSecret = process.env.EVALCANARY_WEBHOOK_SECRET;
if (!webhookSecret) {
  throw new Error("EVALCANARY_WEBHOOK_SECRET is missing");
}

createEvalCanaryEndpoint({
  projectId: "your-project-id",
  webhookSecret,
  agents,
});

EvalCanary signs the exact request bytes with HMAC-SHA256 and sends an EvalCanary-Signature header:

t=1786406400,v1=0123456789abcdef...

The SDK checks the signature and its five-minute timestamp window before parsing the body or calling your agent. Changing any byte invalidates the signature. Mount the SDK handler directly and give it the untouched Fetch Request; do not parse or rebuild the body first.

The endpoint verifies requests locally. It does not send the signing secret back to EvalCanary or contact EvalCanary during request handling.

Roll the signing secret

Choose Roll secret in Project settings when you need a replacement. The new secret is shown once. Deploy it as the new EVALCANARY_WEBHOOK_SECRET within 24 hours.

During that window, EvalCanary signs each request with both the new and previous secrets. The signature header contains one v1 value for each active secret, so either deployed secret verifies the request. After 24 hours, the previous secret is no longer accepted.

Retries and side effects

Every signed body includes a unique requestId. A captured valid request can be replayed during the five-minute signature window, just as a signed webhook can be delivered more than once. Pass request.requestId as the idempotency key for any tool or downstream operation that can change data. Keep production checks pointed at sandboxed or otherwise contained accounts.

Synthetic actors

Use test profiles only when a scenario needs identity-dependent behavior. Define finite actor aliases and fail closed for missing or unknown values:

const actorAliases = {
  "standard-customer": true,
  "vip-customer": true,
} as const;

async function resolveContext({
  actor,
}: {
  actor: keyof typeof actorAliases | undefined;
}) {
  if (!actor) throw new TypeError("A test profile is required");
  return { downstreamHeaders: await headersForSyntheticActor(actor) };
}

Map aliases to sandbox accounts or contained services. Do not accept customer IDs, session values, or database record IDs from scenario input. The context returned by resolveContext stays in application memory unless an agent includes it in output or evidence.

Stored data boundary

Check scenario input is stored with the check version. A completed run can retain each sample's input, authenticated request, raw endpoint response, typed evidence, pass-condition results, error details, and target revision. LLM usage can include provider and model identifiers, provider request IDs, and token counts.

Return only the evidence needed to evaluate behavior. Do not put credentials, cookies, access tokens, or live customer data in check input, output, evidence, tool input, or tool output.

AI judgment evaluates the sample against the configured rubric. Use deterministic content or tool-call pass conditions when sample evidence should not be sent through that evaluator.