> ## Documentation Index
> Fetch the complete documentation index at: https://docs.usebench.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Set up real app testing

> Run the actual entry point, including model decisions, tools and orchestration.

For setup and management entirely from a coding agent, see the [headless workflow](/guides/headless) and [SDK platform clients](/sdk/platform).

For an overview, see [real app testing and simulation testing](/guides/testing-modes).

## Copy this into your coding agent

Paste into Cursor, Codex, Claude Code, or another coding agent in your project.

```text theme={null}
Add real app tests using https://docs.usebench.ai/sdk/system-evaluation.
Find the real app entry point and instrument its model and tool calls with Bench.
Create a small suite using synthetic data and isolated test dependencies.
Check the actual result and tool effects, including one retry or failure case.
Run the suite and show passed, failed and incomplete checks. Do not use live customer data.
```

Real app tests run your app with sample inputs and check what it actually does, including tool calls and the final result. Use them to check that a refund was recorded, a weather tool returned a result, or a workflow reached the right outcome.

**Start benching** compares prompts and models. Real app tests cover the code around them. Connecting a repository alone does not run that application.

Without an application connection, Bench can still run model conversations using
simulated tool responses. It checks the model's choices and final answer. Those
results are labeled as simulations; they do not prove that real tools or database
changes worked. Connect real app testing to check those effects.

All four languages support real app tests and scripted simulations:

| Language                                     | Real app testing  | Simulations       |
| -------------------------------------------- | ----------------- | ----------------- |
| [TypeScript and JavaScript](/sdk/typescript) | `evaluateSystem`  | `simulateSystem`  |
| [Python](/sdk/python)                        | `evaluate_system` | `simulate_system` |
| [Go](/sdk/go)                                | `EvaluateSystem`  | `SimulateSystem`  |
| [Rust](/sdk/rust)                            | `evaluate_system` | `simulate_system` |

The examples below use TypeScript. Each language guide includes its native API and a runnable simulation.

## Run a small suite

Complete [SDK setup](/sdk/quickstart), then run this on a test server, never a live
customer request path. Use synthetic inputs and safe tool dependencies. The SDK
does not sandbox your application or prevent its external side effects.

```ts theme={null}
const report = await bench.evaluateSystem({
  sourceRevision: process.env.BENCH_SOURCE_COMMIT!, // Full 40-character SHA
  contextRevision: 'refund-policy-v1',
  cases: [
    {
      id: 'refund-after-deadline',
      split: 'incident',
      input: { orderId: 'fixture-order', days: 31 },
      expectedOutput: { refunded: false },
      requiredTools: ['load-policy'],
      forbiddenTools: ['issue-refund'],
      maxModelCalls: 3,
    },
  ],
  run: (input, { signal }) => runYourActualApplication(input, { signal }),
})

// Optional. Saving a report does not run a judge or use evaluation credits.
await bench.publishSystemEvaluation(systemId, report)
await bench.shutdown()
```

`runYourActualApplication` is your application's entry point, not a prompt assembled
by Bench. Instrument its nested model, agent and tool calls with `bench.trace`.
Await every call, including completion of streams, before returning the final result.
Return the business outcome you want to check, such as a fixture database's refund
state. A model saying “refunded” is not evidence that a refund actually happened.

## Cases, criteria and coverage

An expected output is an exact typed comparison. Required tools must have a successful
recorded call. Forbidden tools must have no recorded call. Tool/model call budgets
catch unexpected loops. These are deterministic checks, not semantic judgments.

Use suite partitions deliberately:

| Partition  | Purpose                                                         |
| ---------- | --------------------------------------------------------------- |
| Capability | Can the application perform a supported task?                   |
| Regression | Does established behavior still work?                           |
| Incident   | Does a reproduced production failure remain fixed?              |
| Holdout    | Does the change work on independent cases not used to build it? |

The SDK accepts 1 to 100 cases. Each case has at most 90 tool assertions and 100
captured spans. Missing assertions, missing root evidence, capture limits, timeout,
or unfinished child work produce an incomplete result. Timeouts are cooperative;
untrusted or non-cooperative code needs an isolated process or sandbox. The suite
stops after a timeout or unfinished work rather than overlapping the next case.

Reports pin source revision, context revision, suite hash and planned case count.
They retain redacted inputs, expected outcomes, checks and observed trajectories.
Redaction can remove detail needed for a repair; inspect it before sharing a brief.
Only explicitly instrumented behavior is covered. Unrecorded calls are unknown.

## View results

Open **AI systems → your system → Real app testing**. Browse cases and their tool, harness or
quality findings. **Review** shows checks and recorded execution. **Copy fix prompt**
includes the case and pinned revisions for your approved coding agent.

Saved SDK reports are labeled **SDK test report**. They are not a server
certification, a prompt-benchmark score, or proof that a production issue was fixed.
Reports are limited to 500 KB, deduplicated, and retained for 30 days. Bench
accepts up to 100 reports per system during that window; keep additional reports
locally. A repository-scoped key may access only its permitted systems.

MCP exposes these reports through `bench_get_runtime_evaluations`, without an
evaluation charge. It does not execute the customer's application itself.

Next: [run tests in CI](/sdk/ci), [SDK setup](/sdk/quickstart) and [production checks](/sdk/production-checks).

## Simulate a customer conversation

Use `simulateSystem` to replay up to 20 scripted customer turns through a fresh
application session for each case. The session keeps your application's routing,
prompts, tool wrappers and retries. Give its external services test fixtures, then
read those fixtures to check what actually happened.

For example, a repeated request must not refund the same order twice:

```ts theme={null}
const report = await bench.simulateSystem({
  sourceRevision: process.env.GIT_COMMIT_SHA!,
  contextRevision: 'refund-policy-v1',
  cases: [{
    id: 'repeated-refund-request',
    split: 'incident',
    businessOutcome: 'Refund an eligible order exactly once',
    input: {
      initialState: { orderId: 'order-a', refundableCents: 12000 },
      turns: [
        'Please refund order A.',
        'I did not get confirmation. Please refund order A.',
      ],
    },
    expectedState: { refundedCents: 12000, transactions: 1 },
  }],
  createSession: async (initialState, { signal }) => {
    // Your test adapter creates a fresh payment fixture and app session.
    const fixture = await createPaymentFixture(initialState)
    const app = createSupportAgent({ payments: fixture.client })
    return {
      turn: (message) => app.respond(message, { signal }),
      observe: () => fixture.readRefundSummary(),
      close: () => fixture.close(),
    }
  },
})
```

`createPaymentFixture` and `createSupportAgent` are your application's test adapters,
not SDK exports. The observer should read the fixture ledger or test database;
it must not reconstruct state from the final answer. Missing state evidence makes
the result incomplete. An answer saying “refunded $120” can pass a wording check
while a ledger showing $240 fails the outcome check.

The turns are scripted and repeatable. Bench does not generate an adaptive customer
inside this API or automatically clone your external services. Call your real
application entry point from `turn`; replacing its retry wrapper with a mock would
hide retry bugs. Fixture state is reset by creating a new session for every case.

## Keep capability, incident and regression cases together

| Case       | What to check                                                                      |
| ---------- | ---------------------------------------------------------------------------------- |
| Capability | A normal eligible order receives one refund.                                       |
| Incident   | The refund commits, its acknowledgement times out, and retry creates no duplicate. |
| Regression | A timeout before any refund commits can still recover.                             |
| Holdout    | Repeating the customer's request still produces one refund.                        |
| Holdout    | Two different eligible orders both receive their own refunds.                      |

Freeze the same cases, expected state and context revision before comparing changes.
A fix that stops all retries may fix duplication but break recovery. A fix using
one shared idempotency key may break the second legitimate order. Inspect both
failures and previously passing cases before accepting a recommendation.

The SDK records client-provided test evidence. It does not authorize a pull request
or deploy a change. Your application's provider calls may incur provider charges;
producing a local SDK report does not consume Bench evaluations.