Set up real app testing

Run the actual entry point, including model decisions, tools and orchestration.

For setup and management entirely from a coding agent, see the headless workflow and SDK platform clients.

For an overview, see real app testing and simulation testing.

Copy this into your coding agent

Paste into Cursor, Codex, Claude Code, or another coding agent in your project.

Add real app tests using https://docs.usebench.ai/sdk/system-evaluation.
Find the real app entry point and instrument its model and tool calls with Bench.
Create a small suite using synthetic data and isolated test dependencies.
Check the actual result and tool effects, including one retry or failure case.
Run the suite and show passed, failed and incomplete checks. Do not use live customer data.

Real app tests run your app with sample inputs and check what it actually does, including tool calls and the final result. Use them to check that a refund was recorded, a weather tool returned a result, or a workflow reached the right outcome.

Start benching compares prompts and models. Real app tests cover the code around them. Connecting a repository alone does not run that application.

Without an application connection, Bench can still run model conversations using simulated tool responses. It checks the model’s choices and final answer. Those results are labeled as simulations; they do not prove that real tools or database changes worked. Connect real app testing to check those effects.

All four languages support real app tests and scripted simulations:

Language	Real app testing	Simulations
TypeScript and JavaScript	evaluateSystem	simulateSystem
Python	evaluate_system	simulate_system
Go	EvaluateSystem	SimulateSystem
Rust	evaluate_system	simulate_system

The examples below use TypeScript. Each language guide includes its native API and a runnable simulation.

Run a small suite

Complete SDK setup, then run this on a test server, never a live customer request path. Use synthetic inputs and safe tool dependencies. The SDK does not sandbox your application or prevent its external side effects.

const report = await bench.evaluateSystem({
  sourceRevision: process.env.BENCH_SOURCE_COMMIT!, // Full 40-character SHA
  contextRevision: 'refund-policy-v1',
  cases: [
    {
      id: 'refund-after-deadline',
      split: 'incident',
      input: { orderId: 'fixture-order', days: 31 },
      expectedOutput: { refunded: false },
      requiredTools: ['load-policy'],
      forbiddenTools: ['issue-refund'],
      maxModelCalls: 3,
    },
  ],
  run: (input, { signal }) => runYourActualApplication(input, { signal }),
})

// Optional. Saving a report does not run a judge or use evaluation credits.
await bench.publishSystemEvaluation(systemId, report)
await bench.shutdown()

runYourActualApplication is your application’s entry point, not a prompt assembled by Bench. Instrument its nested model, agent and tool calls with bench.trace. Await every call, including completion of streams, before returning the final result. Return the business outcome you want to check, such as a fixture database’s refund state. A model saying “refunded” is not evidence that a refund actually happened.

Cases, criteria and coverage

An expected output is an exact typed comparison. Required tools must have a successful recorded call. Forbidden tools must have no recorded call. Tool/model call budgets catch unexpected loops. These are deterministic checks, not semantic judgments.

Use suite partitions deliberately:

Partition	Purpose
Capability	Can the application perform a supported task?
Regression	Does established behavior still work?
Incident	Does a reproduced production failure remain fixed?
Holdout	Does the change work on independent cases not used to build it?

The SDK accepts 1 to 100 cases. Each case has at most 90 tool assertions and 100 captured spans. Missing assertions, missing root evidence, capture limits, timeout, or unfinished child work produce an incomplete result. Timeouts are cooperative; untrusted or non-cooperative code needs an isolated process or sandbox. The suite stops after a timeout or unfinished work rather than overlapping the next case.

Reports pin source revision, context revision, suite hash and planned case count. They retain redacted inputs, expected outcomes, checks and observed trajectories. Redaction can remove detail needed for a repair; inspect it before sharing a brief. Only explicitly instrumented behavior is covered. Unrecorded calls are unknown.

View results

Open AI systems → your system → Real app testing. Browse cases and their tool, harness or quality findings. Review shows checks and recorded execution. Copy fix prompt includes the case and pinned revisions for your approved coding agent.

Saved SDK reports are labeled SDK test report. They are not a server certification, a prompt-benchmark score, or proof that a production issue was fixed. Reports are limited to 500 KB, deduplicated, and retained for 30 days. Bench accepts up to 100 reports per system during that window; keep additional reports locally. A repository-scoped key may access only its permitted systems.

MCP exposes these reports through bench_get_runtime_evaluations, without an evaluation charge. It does not execute the customer’s application itself.

Next: run tests in CI, SDK setup and production checks.

Simulate a customer conversation

Use simulateSystem to replay up to 20 scripted customer turns through a fresh application session for each case. The session keeps your application’s routing, prompts, tool wrappers and retries. Give its external services test fixtures, then read those fixtures to check what actually happened.

For example, a repeated request must not refund the same order twice:

const report = await bench.simulateSystem({
  sourceRevision: process.env.GIT_COMMIT_SHA!,
  contextRevision: 'refund-policy-v1',
  cases: [{
    id: 'repeated-refund-request',
    split: 'incident',
    businessOutcome: 'Refund an eligible order exactly once',
    input: {
      initialState: { orderId: 'order-a', refundableCents: 12000 },
      turns: [
        'Please refund order A.',
        'I did not get confirmation. Please refund order A.',
      ],
    },
    expectedState: { refundedCents: 12000, transactions: 1 },
  }],
  createSession: async (initialState, { signal }) => {
    // Your test adapter creates a fresh payment fixture and app session.
    const fixture = await createPaymentFixture(initialState)
    const app = createSupportAgent({ payments: fixture.client })
    return {
      turn: (message) => app.respond(message, { signal }),
      observe: () => fixture.readRefundSummary(),
      close: () => fixture.close(),
    }
  },
})

createPaymentFixture and createSupportAgent are your application’s test adapters, not SDK exports. The observer should read the fixture ledger or test database; it must not reconstruct state from the final answer. Missing state evidence makes the result incomplete. An answer saying “refunded $120” can pass a wording check while a ledger showing$240 fails the outcome check.

The turns are scripted and repeatable. Bench does not generate an adaptive customer inside this API or automatically clone your external services. Call your real application entry point from turn; replacing its retry wrapper with a mock would hide retry bugs. Fixture state is reset by creating a new session for every case.

Keep capability, incident and regression cases together

Case	What to check
Capability	A normal eligible order receives one refund.
Incident	The refund commits, its acknowledgement times out, and retry creates no duplicate.
Regression	A timeout before any refund commits can still recover.
Holdout	Repeating the customer’s request still produces one refund.
Holdout	Two different eligible orders both receive their own refunds.

Freeze the same cases, expected state and context revision before comparing changes. A fix that stops all retries may fix duplication but break recovery. A fix using one shared idempotency key may break the second legitimate order. Inspect both failures and previously passing cases before accepting a recommendation.

The SDK records client-provided test evidence. It does not authorize a pull request or deploy a change. Your application’s provider calls may incur provider charges; producing a local SDK report does not consume Bench evaluations.