> ## Documentation Index
> Fetch the complete documentation index at: https://docs.usebench.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Understand an evaluation

> From evidence and criteria to results and proposed fixes.

## Copy this into your coding agent

Paste into Cursor, Codex, Claude Code, or another coding agent in your project.

```text theme={null}
Help me run and understand a Bench using https://docs.usebench.ai/guides/evaluations.
Use Bench MCP to identify the system, inspect context and check my evaluation allowance.
Explain what the run will test and ask before starting it.
Then follow progress and show failed checks, unscored cases, cost and proposed fixes.
Distinguish prompt results from evidence that the application and its tools actually ran.
```

<Note>Failed or canceled evaluations return their reserved allowance.
Automatic retries share the original reservation. A Free whole-system bench uses
one allowance only after every prompt finishes successfully. A completed score
of zero still counts as a successful evaluation, because Bench measured a failure.
Reading results and fully cached replays do not use another allowance.</Note>

Compare [real app testing and simulation testing](/guides/testing-modes) to see what each run can verify.

## The evaluation path

1. **Pin inputs.** New benches save a redacted source-prompt snapshot before any model call, alongside the selected context revision and evaluation configuration. Dynamic values remain marked unresolved; the source preview is not a recording of every rendered runtime message.
2. **Build criteria.** Generated criteria describe observable good and bad behavior. Explicit library criteria are included without asking the model to rewrite them.
3. **Route checks.** Checks use deterministic evaluation or an appropriate judge. Unsupported or insufficiently grounded checks are excluded from scoring.
4. **Build cases.** Selected golden cases seed the suite. Generated cases cover remaining needs within the plan budget.
5. **Measure the baseline.** Bench executes the prompt and evaluates the response against the suite. Provider failures and unsupported replay inputs remain unscored.
6. **Compare candidates.** Candidate prompt/model combinations are measured against the same suite. Primary and secondary criteria both count. Breaking an individually passing baseline check blocks a recommendation, even if an average improves. Reported quality and cost come from those results.
7. **Review a recommendation.** The suggested change is available for investigation. Applying a patch, opening a PR, merging and deploying are distinct actions.

## Investigate failures

**History** lists one row per bench. **Reports** shows benches with completed prompt results. Each row shows quality change, estimated cost change and proposed models, with ten benches per page. Open **View details** to inspect a prompt, then switch between prompts in the same bench without leaving the panel. Findings, attention cards and suggested fixes in Feed also open their details on the right, keeping the list in place.

The investigation panel has five tabs:

| Tab          | What you see                                                          |
| ------------ | --------------------------------------------------------------------- |
| Activity     | Saved stages, model and test cases                                    |
| Test results | Criteria, passed and failed checks, cases and feedback                |
| Observations | Context gaps and findings that could not become scored criteria       |
| Errors       | Grouped execution problems with affected-case counts                  |
| Fixes        | Source and proposed prompts, model changes and a coding-agent handoff |

Older runs without a source snapshot say so; Bench does not substitute today's source for a historical prompt. Deleted evidence stays unavailable. Close the panel to return to the results or history you were viewing. Reports can export the selected prompt's saved result as JSON.

Newly executed baselines can also include **Evidence review** under Observations.
This is a second AI opinion on up to five cases, using their recorded responses
and matching criteria. The display states how many cases were reviewed.
It does not rerun the app, change scores, or fill in missing tool results.
Older or cached baselines may not have this additional review.

Observations use a paginated table. Filter by **Needs context** or **To review**, then use **View details** to open the supporting explanation in the right sidebar. These observations are not confirmed quality failures. Saved cases separate **Input**, **Expected result** and **Checks** with clear labels.

Add feedback when a test or judgment is wrong. Specify whether a correction belongs to that case or to a general system rule. To edit future cases, use [Test library](/guides/test-library).

History charts show quality, failing checks or model cost. They join only comparable runs with matching prompt and evaluation provenance. A candidate is not a production improvement. Changes to criteria or context can make scores incomparable.

<Warning>
  Testing a candidate on the suite used to optimize it is not held-out validation or proof of causality in production. Use independent cases and runtime evidence before rolling out a consequential change.
</Warning>

## Quality and cost

The **Bench score** in a system's Results is the average baseline quality score across prompts in a completed bench, shown out of 100. Quality and estimated cost changes are shown separately. No score is invented when scoring failed or is incomplete.

* **Quick scan:** source-backed risks and price comparisons, not executed behavioral tests. "Quality unverified" is not a failing quality score. "Potentially lower model cost" assumes the same input and output volume, not measured overpayment. Published model-price comparisons expire after 30 days unless refreshed. See [quick-scan impact](/guides/quick-scan).
* **Completed bench:** the candidate is compared with the original setup on the same suite. Quality changes are percentage-point differences, not a claim about production improvement.
* **Cost impact:** estimated model cost per test call. A multi-prompt bench uses an unweighted average across its prompts, not their production traffic mix or monthly bill. Missing or zero price coverage does not become a 100% saving. A cheaper candidate that regresses a prompt is flagged for quality review.
* **Proposed change:** a recommendation, not a merged fix. No production issue is marked solved just because a candidate scored better.

Filter findings by quality or cost impact, or by component: prompts, models, tools,
harness/workflow or system. The **Quality and cost checks** card toggles scan
findings on or off. Scan findings, evaluation findings, criteria and cases are
paginated so large suites remain navigable.

The Feed's **Quality increase** and **Cost decrease** cards summarize measured
comparisons from the latest completed bench for each system. Missing comparisons
show a quiet empty state rather than an invented score. These are candidate test
results, not deployed improvements or monthly savings.

## While Bench works

One **Bench is working** panel contains the stages for the whole system. Expand a stage to see the prompts being tested, their activity and saved details. Concurrent prompts may occupy different stages. Prompt rows are paginated. Leaving the browser does not stop the server-side evaluation.

## If a run stops

The Results page shows one warning for the latest stopped bench, with one **Retry bench** action when allowance remains. Close the notice to hide it for the current browser session, including page navigation and refresh. No restore button replaces it. When the allowance is exhausted, **Upgrade to bench more** opens the plan offer before any new evaluation starts. Older failures remain in History. A stopped run is not a quality failure and does not have a measured candidate impact.

A provider credit or billing error needs the model-provider account resolved before retrying. Upgrading a Bench plan does not replenish a model provider's balance. Saved source evidence remains visible even if scoring never starts.

The app keeps operational diagnostics private and offers a concise error and **Contact us** action. In the Feed, close an attention notice to hide it for the current browser session. When no notices remain, the attention section disappears. This does not resolve or delete the issue, and a new stopped run produces a new notice.

Worker leases prevent stale workers from overwriting terminal results and recover
stale reservations. An interrupted execution can still require a new run.
Context-summary and production-check jobs have separate durable queues.

## Real app testing

Use the [real app test SDK](/sdk/system-evaluation) to execute your
actual entry point, tools and orchestration against capability, regression,
incident and independent holdout cases. SDK results are explicitly client-reported
and appear under **Real app testing**, separate from prompt Bench scores. The normal Start
benching action does not yet configure a hosted runner for arbitrary repositories.

## When no change qualifies

Bench still provides a next improvement to investigate. A proposed prompt edit is
labeled **needs testing** until it passes the comparison. A better average does
not qualify a change that breaks regression checks or leaves important evidence
unresolved. If the current cases already pass, add a recent incident or another
business task to broaden the next comparison.

Model comparisons use fixed simulated tool replies when tools are declared.
These results measure model behavior and model-call cost. They do not establish
that application code or persistent side effects worked. Use
[real app tests](/sdk/system-evaluation) for that evidence.