> ## Documentation Index
> Fetch the complete documentation index at: https://docs.usebench.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Production checks

> Compare recorded interactions with saved criteria.

## Copy this into your coding agent

Paste into Cursor, Codex, Claude Code, or another coding agent in your project.

```text theme={null}
Help me check a recorded interaction using https://docs.usebench.ai/sdk/production-checks.
Verify SDK delivery with synthetic staging traffic and inspect the saved criteria.
Explain what content capture is needed before changing the metadata-only default.
Keep production capture and automatic checks off. Prepare one check and show its
evaluation cost before I start it. Explain the evidence behind the result.
```

Open an AI system's **Production** tab. The centered **Set up SDK** and **Quickstart**
actions help you connect application activity. Reading the short monitoring introduction
does not enable capture. Recorded input/output token counts appear when the
SDK sends them; missing usage is not treated as zero or a verified cost saving.

## Capture first, evaluate separately

Metadata-only capture is free of evaluation-credit charges. Inputs and outputs
are omitted by default. To evaluate real behavior, explicitly enable content
capture, supply the span's `input`, and add a project-specific `redact` function.
In TypeScript, the traced callback result becomes the output. In Python and manual
span APIs, record the result explicitly using the language guide.
Only send data you have permission to process.

```ts theme={null}
const bench = new Bench({
  apiKey: process.env.BENCH_API_KEY!,
  repository: 'your-team/your-app',
  branch: 'main',
  captureContent: true,
  redact: removeYourDomainIdentifiers,
})
```

Built-in redaction catches common secrets and identifiers, but does not guarantee
anonymization of free text. Raw traces and their linked production-check payloads
are deleted from the application database after 30 days. This is not a claim
about backup deletion or every other category of Bench data. Intentionally saved
datasets and cases have their own lifecycle.

## Run a check

1. Generate a rubric by benching a connected prompt.
2. Open a recorded interaction under **Production** and link it to that prompt.
3. Choose **Check interaction**, then confirm the use of one evaluation.

The API responds with a queued job immediately. Workers pin the rubric content
and digest, reserve one shared account credit, run the judge, then save results.
You can leave the page and return. Results show failed checks and their reasons.
Identical evidence is deduplicated. Failed checks can be retried explicitly.
Interrupted jobs are failed after their lease expires; reserved credits are
reconciled. Failed execution returns the account credit, but attempted execution
still counts toward a key's safety cap.

Automatic checks are off by default. Enable them in the system's **Production**
tab and choose its event environment. Events must be linked to a prompt with saved
criteria. The sending key needs evaluation allowance; the default SDK setup key
cannot spend evaluations. Existing continuous checks for a linked prompt also apply.

## Turn feedback into better benchmarks

Passed checks provide regression examples. Failed checks provide incident cases.
Bench uses their recorded inputs and existing criteria. A recorded answer never
becomes the expected answer just because the application produced it. Incomplete
or uncertain checks do not become passing examples.

Enable **Automatically investigate failures** to start a new Bench after a failed
check, using both recent failures and passing examples from the same environment.
This is limited to one automatic Bench per system per day, requires available
account allowance, and waits while another Bench is running. The previous rubric,
business context revision and original cases stay pinned for the comparison.

The new Bench compares prompts and models using fixed simulated tool replies.
Available recorded tool results preserve the incident's conditions for that replay.
Real app tests are a separate step that runs your actual app to verify effects.
Original regression cases stay in the suite; new production examples fit the
remaining case allowance, with the triggering failure first.
It does not deploy code, publish a pull request, or silently rewrite business
policy. Review tool and harness suggestions with real app tests before shipping.
You can also save reviewed cases or criteria through the test library.

### Check a recorded execution

API and MCP callers can explicitly set `evidence_scope: "trace"` and select the
completed root span. Bench validates one connected, finished tree and includes
its recorded model/tool relationships, inputs, outputs, status and attributes.
Incomplete trees or evidence over 45 KB are rejected, not silently shortened.
The default scope remains `span`.

This checks the recorded interaction against the linked prompt's criteria. It does
not run the application again. Calls that were not recorded cannot be checked.
Use [real app tests](/sdk/system-evaluation) to repeat a scenario with your app.

## Delivery and storage

Exports are buffered and bounded, retried once, and dropped if delivery fails.
Use `onError` and `bench.stats` to monitor drops. In serverless applications,
`await bench.flush()` in a supported lifecycle hook before the process freezes.
Use `shutdown()` after in-flight requests finish at process exit, not per request
on a reusable instance. The SDK is not a durable spool.

Bench stores the traces you send. Monitor export errors in your application so you
can tell when an event did not reach Bench. See [retention and deletion](/guides/retention-and-deletion).