Production checks

Compare recorded interactions with saved criteria.

Copy this into your coding agent

Paste into Cursor, Codex, Claude Code, or another coding agent in your project.

Help me check a recorded interaction using https://docs.usebench.ai/sdk/production-checks.
Verify SDK delivery with synthetic staging traffic and inspect the saved criteria.
Explain what content capture is needed before changing the metadata-only default.
Keep production capture and automatic checks off. Prepare one check and show its
evaluation cost before I start it. Explain the evidence behind the result.

Open an AI system’s Production tab. The centered Set up SDK and Quickstart actions help you connect application activity. Reading the short monitoring introduction does not enable capture. Recorded input/output token counts appear when the SDK sends them; missing usage is not treated as zero or a verified cost saving.

Capture first, evaluate separately

Metadata-only capture is free of evaluation-credit charges. Inputs and outputs are omitted by default. To evaluate real behavior, explicitly enable content capture, supply the span’s input, and add a project-specific redact function. In TypeScript, the traced callback result becomes the output. In Python and manual span APIs, record the result explicitly using the language guide. Only send data you have permission to process.

const bench = new Bench({
  apiKey: process.env.BENCH_API_KEY!,
  repository: 'your-team/your-app',
  branch: 'main',
  captureContent: true,
  redact: removeYourDomainIdentifiers,
})

Built-in redaction catches common secrets and identifiers, but does not guarantee anonymization of free text. Raw traces and their linked production-check payloads are deleted from the application database after 30 days. This is not a claim about backup deletion or every other category of Bench data. Intentionally saved datasets and cases have their own lifecycle.

Run a check

1. Generate a rubric by benching a connected prompt.
2. Open a recorded interaction under Production and link it to that prompt.
3. Choose Check interaction, then confirm the use of one evaluation.

The API responds with a queued job immediately. Workers pin the rubric content and digest, reserve one shared account credit, run the judge, then save results. You can leave the page and return. Results show failed checks and their reasons. Identical evidence is deduplicated. Failed checks can be retried explicitly. Interrupted jobs are failed after their lease expires; reserved credits are reconciled. Failed execution returns the account credit, but attempted execution still counts toward a key’s safety cap.

Automatic checks are off by default. Enable them in the system’s Production tab and choose its event environment. Events must be linked to a prompt with saved criteria. The sending key needs evaluation allowance; the default SDK setup key cannot spend evaluations. Existing continuous checks for a linked prompt also apply.

Turn feedback into better benchmarks

Passed checks provide regression examples. Failed checks provide incident cases. Bench uses their recorded inputs and existing criteria. A recorded answer never becomes the expected answer just because the application produced it. Incomplete or uncertain checks do not become passing examples.

Enable Automatically investigate failures to start a new Bench after a failed check, using both recent failures and passing examples from the same environment. This is limited to one automatic Bench per system per day, requires available account allowance, and waits while another Bench is running. The previous rubric, business context revision and original cases stay pinned for the comparison.

The new Bench compares prompts and models using fixed simulated tool replies. Available recorded tool results preserve the incident’s conditions for that replay. Real app tests are a separate step that runs your actual app to verify effects. Original regression cases stay in the suite; new production examples fit the remaining case allowance, with the triggering failure first. It does not deploy code, publish a pull request, or silently rewrite business policy. Review tool and harness suggestions with real app tests before shipping. You can also save reviewed cases or criteria through the test library.

Check a recorded execution

API and MCP callers can explicitly set evidence_scope: "trace" and select the completed root span. Bench validates one connected, finished tree and includes its recorded model/tool relationships, inputs, outputs, status and attributes. Incomplete trees or evidence over 45 KB are rejected, not silently shortened. The default scope remains span.

This checks the recorded interaction against the linked prompt’s criteria. It does not run the application again. Calls that were not recorded cannot be checked. Use real app tests to repeat a scenario with your app.

Delivery and storage

Exports are buffered and bounded, retried once, and dropped if delivery fails. Use onError and bench.stats to monitor drops. In serverless applications, await bench.flush() in a supported lifecycle hook before the process freezes. Use shutdown() after in-flight requests finish at process exit, not per request on a reusable instance. The SDK is not a durable spool.

Bench stores the traces you send. Monitor export errors in your application so you can tell when an event did not reach Bench. See retention and deletion.