Python

Record Python agent, model and tool activity in Bench.

Copy this into your coding agent

Paste into Cursor, Codex, Claude Code, or another coding agent in your project.

Add Bench using https://docs.usebench.ai/sdk/python and the setup key from my Bench SDK page. Keep the key server-side. Label the
environment and start with metadata-only capture. Wrap one real request and
its tools, including async context propagation. Flush a synthetic interaction
and confirm it arrived in Bench. Show changed files and remaining setup steps.

Beta release. All four languages support tracing, real app tests and scripted simulations.

Install

Package on PyPI · Source and examples

python -m pip install trybench-sdk==0.1.0

Record one interaction

Choose Python on the SDK setup page and copy the coding-agent prompt. Bench fills in your account setup key automatically. Python 3.10 or later is required.

import os
from bench_sdk import Bench

bench = Bench(
    api_key=os.environ["BENCH_API_KEY"],
    repository=os.environ["BENCH_REPOSITORY"],
    branch=os.environ["BENCH_BRANCH"],
    endpoint=os.environ["BENCH_API_BASE_URL"],
    system_name="Support agent",
    environment="staging",
)

with bench.trace("connection-check", kind="AGENT") as span:
    span.set_output({"ok": True})

bench.shutdown()

Open the system’s Production tab. Setup changes from Waiting for first event to SDK events received after the event is accepted.

Wrap your application

Use with bench.trace(...) around a Deep Agents, LangGraph, LangChain or custom application call. It also works around await agent.ainvoke(...). Wrap tools inside the same request with kind="TOOL"; nested async tasks inherit their parent trace. Keep streaming spans open until the stream finishes.

Inputs and outputs are omitted by default. With content capture enabled, pass input=... and call span.set_output(result). Built-in filtering runs before delivery. Add a redact(value) callback for your application’s extra fields. See privacy and redaction.

Call bench.flush() at request boundaries, or await bench.aflush() in async code. Call shutdown() after active requests finish. Delivery retries once; bench.stats shows queued and dropped spans. A delivery failure does not change your application’s result or exception.

Set component_id to a real prompt component from Bench to connect the event to its saved criteria. Production checks require recorded content and a key with evaluation allowance.

Automatic framework adapters are coming soon. Use explicit wrappers with your current framework.

Test your application

await bench.evaluate_system(...) calls your application’s request handler with pinned cases. It captures the real nested tool/model traces, compares the final output and independently observed state, and returns a redacted report locally. Use a test database and test service credentials.

report = await bench.evaluate_system(
    source_revision=os.environ["GIT_COMMIT_SHA"],  # Full 40-character commit SHA
    context_revision="refund-policy-v1",
    cases=[{
        "id": "outside-refund-policy", "split": "regression",
        "input": {"days": 45}, "expected_output": {"refunded": False},
        "forbidden_tools": ["issue-refund"],
    }],
    run=lambda request, context: handle_request(request),
)
assert report["summary"]["status"] == "completed"
assert all(case["status"] == "passed" for case in report["cases"])
# Explicit upload, only when you want this report saved in Bench:
await bench.publish_system_evaluation(int(os.environ["BENCH_SYSTEM_ID"]), report)

run(input, context) and observe(context) may be synchronous or asynchronous. When a case has expected_state, supply observe to read the authoritative test state. Context provides case_id, cancelled, signal, deadline and raise_if_cancelled(). Inputs are JSON snapshots; changes inside the application do not change the case’s assertions.

await bench.simulate_system(...) accepts the same revisions and cases, plus create_session(initial_state, context). Cases use input={"initial_state": {...}, "turns": [...]} and require expected_state. Return an object with turn(message, context), observe() and close() methods. A fresh session receives 1 to 20 scripted customer turns. Bench snapshots observed state before closing the session. context.turn_index identifies the turn.

Both helpers use a 30-second timeout per case, configurable with timeout up to 300 seconds. Pass a threading.Event as cancel_event to stop the suite. Missing assertions, missing state, unfinished traces and timeouts remain incomplete. Callbacks must honor cancellation; Python cannot forcibly stop a synchronous thread. Await all child work and isolate external side effects. These helpers are local execution, not a process sandbox or a hosted verification claim.

Tests record redacted content even when production capture is metadata-only, so use synthetic inputs. Reports are not uploaded and paid checks are not started unless you take a separate explicit action. Automatic framework adapters are coming soon.

Run the complete refund simulation example. It sends two refund requests, checks the number of refunds written, and demonstrates the failing implementation and its fix.

Run in CI

Use Bench with your existing pull-request checks. See Run tests in CI for setup, report handling and the features available in each language.