Run tests in CI

Check application outcomes on pull requests and stop regressions before merging.

Copy this into your coding agent

Paste into Cursor, Codex, Claude Code, or another coding agent in your project.

Set up Bench in CI using https://docs.usebench.ai/sdk/ci.
Use my existing SDK installation and repository-scoped key. Keep keys in CI secrets.
Use the real app test or simulation helper for this project's language. Run
the real application with isolated test dependencies. Include capability, regression
and incident cases. Assert actual tool effects, not just the final answer.
Fail the job for a failed, errored, unscored, missing or incomplete case. Save the
redacted report even when checks fail and flush telemetry before exit.
Keep the existing test runner and make failed or incomplete Bench reports fail
the job. Use the native helper names from the matching language guide.
Use synthetic data and show a passing run, a deliberately failing run, and the
setup still required in my CI provider. Do not expose secrets to forked PR code.

CI runs your tests automatically when someone opens or updates a pull request. Bench can record these runs and check application outcomes in all four languages. Installing the SDK does not create a CI workflow or a required PR check.

Language	Available today
JavaScript / TypeScript	Tracing, real app tests and scripted simulations.
Python	Tracing, real app tests and scripted simulations.
Go	Tracing, real app tests and scripted simulations.
Rust	Tracing, real app tests and scripted simulations.

Prepare the test suite

Complete your language setup and commit its dependency lockfile. Use the public installation command from your language guide. Keep the same SDK version locally and in CI.

For JavaScript and TypeScript, create the cases and application adapter described in real app testing. Keep the same cases, business criteria and test data when comparing a change. Include at least one normal task, one reproduced failure and one previously working behavior.

The following ESM script expects your tests/bench-suite.mjs to export createBenchSuite(bench). That is your application adapter, not an SDK export. It returns contextRevision, cases, run, and, when needed, observe and timeoutMs. Pass the same Bench instance to your instrumented application.

Save as scripts/bench-ci.mjs:

import { mkdir, writeFile } from 'node:fs/promises'
import { execFileSync } from 'node:child_process'
import { Bench } from '@benchai/sdk'
import { createBenchSuite } from '../tests/bench-suite.mjs'

for (const name of ['BENCH_API_KEY', 'BENCH_REPOSITORY', 'BENCH_BRANCH']) {
  if (!process.env[name]) throw new Error(`Set ${name} in CI`)
}

const bench = new Bench({
  apiKey: process.env.BENCH_API_KEY,
  repository: process.env.BENCH_REPOSITORY,
  branch: process.env.BENCH_BRANCH,
  endpoint: process.env.BENCH_API_BASE_URL,
  environment: 'ci',
  captureContent: false,
})

try {
  const suite = await createBenchSuite(bench)
  const report = await bench.evaluateSystem({
    ...suite,
    sourceRevision: execFileSync('git', ['rev-parse', 'HEAD'], {
      encoding: 'utf8',
    }).trim(),
  })

  await mkdir('bench-results', { recursive: true })
  await writeFile('bench-results/report.json', JSON.stringify(report, null, 2))

  const passed = report.summary.status === 'completed'
    && report.planned_case_count > 0
    && report.cases.length === report.planned_case_count
    && report.cases.every((item) => item.status === 'passed')

  console.log(JSON.stringify(report.summary))
  if (!passed) process.exitCode = 1
} finally {
  await bench.shutdown()
}

For scripted conversations, call bench.simulateSystem with your session adapter instead. The report and pass/fail gate have the same shape. A failed check can still have summary.status: 'completed', so checking that field alone is not enough.

Add a GitHub Actions job

Set BENCH_API_KEY as a repository secret and BENCH_API_BASE_URL as a repository variable using the values from Bench. The default SDK key is sufficient for tracing and saving application reports. These operations do not consume Bench evaluations. Your application’s model calls can still have provider costs.

Save as .github/workflows/bench.yml, alongside your existing tests:

name: Bench real app tests
on:
  pull_request:
  workflow_dispatch:

permissions:
  contents: read

jobs:
  application-tests:
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: npm
      - run: npm ci
      # Add your app's build and test-dependency setup here if needed.
      - name: Check application outcomes
        run: node scripts/bench-ci.mjs
        env:
          BENCH_API_KEY: ${{ secrets.BENCH_API_KEY }}
          BENCH_API_BASE_URL: ${{ vars.BENCH_API_BASE_URL }}
          BENCH_REPOSITORY: ${{ github.repository }}
          BENCH_BRANCH: ${{ github.head_ref || github.ref_name }}
      - name: Save the report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: bench-results
          path: bench-results/report.json
          if-no-files-found: warn
          retention-days: 7

The source revision records the commit actually checked out by CI. On a pull request, that can be GitHub’s test merge commit. Add this job as a required check in your repository rules if failed outcomes should block merging.

GitHub does not supply repository secrets to forked pull requests or Dependabot runs. This script fails clearly when the key is missing. Use a reviewed, trusted run for those contributions; never switch to pull_request_target to execute untrusted PR code with secrets. See GitHub’s secrets documentation.

Python, Go and Rust

Call the native real app test or simulation helper from your existing test runner. Read actual test database or tool state, assert that the report passed, and save the redacted report as a CI artifact. Use the same cases and revisions when comparing a change. Set the client environment to ci.

Language	Test command	Setup
Python	python -m unittest discover or your existing pytest command	Python
Go	go test -race ./...	Go
Rust	cargo test --locked	Rust

For Python, require report["summary"]["status"] == "completed" and every case to have status == "passed". In Go, require report.Passed(); in Rust, require report.passed(). Check errors returned by the helper before accepting a report. A failed, cancelled, timed-out or incomplete suite must fail the job.

You can also keep ordinary tracing around existing tests. Flush those traces before exit and inspect delivery statistics separately from test assertions.

Verify the setup

1. Run a correct fixture and confirm the job passes.
2. Change one expected outcome and confirm the job fails with a saved report.
3. Remove an assertion or force a timeout and confirm incomplete work cannot pass.
4. Restore the case, rerun, and inspect the saved report artifact.

To display a report under Real app testing, explicitly publish it with your language’s report helper and the correct system ID. Uploading an Actions artifact alone does not publish it to Bench. See real app testing for the language guides, report limits and examples.

Real app test helpers capture spans inside the report in all four languages. Flushing does not upload those spans as production events. Ordinary tracing outside the helper sends events you can find in Bench’s ci environment.