# Agent quickstart Source: https://docs.benchflow.ai/agent-quickstart # Agent quickstart prompt Want an AI coding agent to set BenchFlow up for you? Copy the entire block below and paste it into your agent (Claude Code, Codex CLI, Gemini CLI, etc.) in an empty working directory. It walks the agent through installing benchflow, running one real benchmark task in a Docker sandbox, inspecting every artifact the run produces, and authoring a task of your own — expect roughly 10–20 minutes end to end, most of it the live eval. The prompt uses generic `/` placeholders; substitute any supported provider. It requires benchflow 0.6.0 or newer. ```text theme={null} You are setting up benchflow, an open-source harness for benchmarking AI coding agents in sandboxes, and running one real evaluation end to end. Work through the numbered steps in order. Show the user each command before running it. GUARDRAILS (apply to every step): - Never print, echo, log, or commit API keys. Never write keys into any file except a local .env (if the working directory is a git repo, confirm .env is gitignored; never commit it anywhere). - Every `bench eval create` invocation — including retries — must use a FRESH --jobs-dir. Reusing a jobs dir triggers resume logic and skips the run. - A `[FAIL]` line with a fractional reward (e.g. reward 0.4, Score: 0/1) means the pipeline is HEALTHY: the agent ran and the verifier scored it below the pass threshold. Only treat a run as broken if step 5's checks fail. - Never delete a jobs directory or any results. If you need a clean slate, make a new directory. - Do not use the `timeout` command (absent on macOS). benchflow has its own agent/verifier timeouts. If any single step stalls with no new output for ~15 minutes, stop it and report what happened instead of waiting forever. STEP 0 — Preflight - Check Docker: `docker info > /dev/null 2>&1`. If it fails, ask the user to start Docker Desktop / the Docker daemon, then re-check. Do not continue without it: benchflow has no up-front daemon check, so a dead daemon fails the run partway through instead of at startup. - Check uv: `command -v uv`. If missing, install it: `curl -LsSf https://astral.sh/uv/install.sh | sh` and ensure it is on PATH. - Python 3.12+ is required; uv will provision it if needed. STEP 1 — Install benchflow This prompt requires benchflow 0.6.0 or newer (the task.md authoring CLI and trainer artifacts shipped in 0.6.0). Install it from PyPI: uv tool install benchflow benchflow pins a stable litellm (no `--prerelease` flag needed). If uv reports "Executables already exist: bench, benchflow", rerun the same command with `--force`. Confirm with `bench --version` after install. If the installed version is still older than 0.6.0, CONTINUE anyway in degraded mode: steps 0–6 work on 0.5.x too. Tell the user which version you got, then (a) in step 6, expect only `trainer/verifiers.jsonl` (the `atif.json`/`adp.jsonl` trainer artifacts ship in 0.6.0), and (b) in step 7, scaffold with plain `bench tasks init my-first-task` (the `--format task-md` flag and `task.md` scaffold are 0.6.0+; older builds emit the split `task.toml` layout — the oracle run works the same). Note in your final summary that upgrading to 0.6.0 unlocks the full flow. TRAP — this quickstart uses the tool-installed `bench` (on PATH after `uv tool install`); if you instead invoke benchflow as `uv run bench …` from a benchflow source checkout, run it from INSIDE that project directory, because `uv run bench` launched from outside the repo can resolve a different or legacy `bench`. Verify: bench --version bench agent list Show the user the version and the agent list. STEP 2 — Fetch one sample task (sparse checkout, not a full clone) The skillsbench repo is large; download only one task: git clone --depth 1 --filter=blob:none --sparse https://github.com/benchflow-ai/skillsbench cd skillsbench && git sparse-checkout set tasks/tictoc-unnecessary-abort-detection && cd .. Confirm the task directory exists and briefly summarize its instruction.md to the user so they know what the benchmark agent will be asked to do. STEP 3 — Set credentials Ask the user which model provider to use, then have them put keys in a local .env and load it with the export-all pattern: set -a; source .env; set +a or export explicitly. If the working directory is a git repo, first confirm .env is gitignored (add it to .gitignore if not); in a fresh non-git directory there is nothing to check — just keep the file local and never print its contents. Provider-prefix rule, in one line: for a model named `/` with a user-supplied endpoint (deepseek, glm, kimi, minimax, hunyuan, ...), benchflow reads `_API_KEY` plus `_BASE_URL`; fixed-endpoint providers (openai, anthropic, gemini, zai, ...) need only the API key. Example for a deepseek-hosted model: export DEEPSEEK_API_KEY=... # never echo this export DEEPSEEK_BASE_URL=https://api.deepseek.com Variables must be EXPORTED (a plain `source .env` without `set -a` never reaches the bench process). If the base URL is missing, the run fails with an explicit "requires DEEPSEEK_BASE_URL" error — that is your hint, not a bug. Note: `bench agent list` may show an agent (e.g. openhands) as requiring LLM_API_KEY. That is the agent's internal setting name, not a variable you set: benchflow maps your provider-prefixed variables to the agent's LLM_* settings automatically. STEP 4 — Run the eval in a Docker sandbox Use a fresh, timestamped jobs dir and concurrency 1: JOBS_DIR="jobs/quickstart-$(date +%Y%m%d-%H%M%S)" bench eval create \ --tasks-dir skillsbench/tasks/tictoc-unnecessary-abort-detection \ --agent openhands \ --model / \ --sandbox docker \ --concurrency 1 \ --jobs-dir "$JOBS_DIR" Substitute the user's chosen model (e.g. an openhands-compatible provider-prefixed model). First run pulls/builds the task image, so expect several minutes. Stream the output to the user. If you retry for any reason, mint a NEW $JOBS_DIR first. STEP 5 — Verify the run is REAL, then explain the score Exit code 0 only means the pipeline completed — it is NOT pass/fail. Find the rollout result: find "$JOBS_DIR" -name result.json A run counts as REAL only if ALL THREE hold in the rollout's result.json: 1. `n_tool_calls` > 0 (the agent actually acted) 2. `agent_result.total_tokens` > 0 (real model traffic was captured) 3. `rewards` is present and its value is not null (the verifier scored it) Check them with a one-liner (kept on one line so indentation cannot break it): python3 -c "import json,sys; d=json.load(open(sys.argv[1])); t=d.get('n_tool_calls') or 0; k=(d.get('agent_result') or {}).get('total_tokens') or 0; r=d.get('rewards'); print('n_tool_calls:',t,'total_tokens:',k,'rewards:',r); print('REAL run' if t>0 and k>0 and r else 'NOT a real run'); sys.exit(0 if t>0 and k>0 and r else 1)" "$(find "$JOBS_DIR" -name result.json | head -1)" Then explain the semantics to the user: `reward` (also in verifier/reward.txt) is the raw verifier value 0.0–1.0; `Score: x/1` and `[PASS]/[FAIL]` apply the pass threshold, where only reward 1.0 counts as a pass. So `[FAIL]` with reward 0.6 means everything worked and the benchmarked agent partially solved the task. If a check in 1–3 fails, report which one and the error/error_category fields from result.json instead of declaring success. STEP 6 — Showcase the artifacts Print the rollout directory tree and tell the user where each artifact landed and what it is: $JOBS_DIR//__/ result.json — rollout summary: rewards, tool calls, token usage/cost, errors, timing config.json — the rollout's resolved configuration (secret-bearing env vars filtered out) prompts.json — the prompts sent to the agent rewards.jsonl — reward record for this rollout timing.json — per-phase timing breakdown agent/ — agent-side logs trajectory/acp_trajectory.jsonl — the full agent trace (every ACP event: prompts, tool calls, outputs) trajectory/llm_trajectory.jsonl — raw provider requests/responses captured by the usage-tracking proxy trainer/verifiers.jsonl — trainer-ready scored trajectory record trainer/atif.json — the trajectory in ATIF interchange format (omitted if the trajectory is empty) trainer/adp.jsonl — the trajectory in ADP format verifier/reward.txt — raw verifier reward verifier/test-stdout.txt — verifier stdout (and ctrf.json when the test emits a CTRF report) Also note the job-level summary.json and the aggregated verifiers.jsonl / adp.jsonl in the job directory. The trainer/ files and the job-level aggregates are written by benchflow 0.6.0+; if they are missing, re-check `bench --version` before reporting a bug. `cost` in result.json can be null for user-endpoint providers (cost telemetry is unavailable for them) — that is a telemetry gap, not a failed run. Show the user one or two sample lines from acp_trajectory.jsonl so they see what a trace looks like. STEP 7 — Author your own task and verify it with the oracle Scaffold a task (in 0.6.0+ `bench tasks init` defaults to the unified task.md format): bench tasks init my-first-task This creates tasks/my-first-task/ with task.md (YAML frontmatter + prompt body), environment/Dockerfile, verifier/test.sh, verifier/test_outputs.py, verifier/verifier.md, verifier/rubrics/, and oracle/solve.sh. The scaffold deliberately fails until edited — test.sh writes reward 0.0, test_outputs.py contains a failing placeholder test, and oracle/solve.sh exits 1 — so an unedited task can never pass by accident. With the user: replace the prompt placeholder in task.md with a small concrete goal — use an ABSOLUTE path so the oracle and verifier agree on location (the agent workspace is `/app`), e.g. "create `/app/hello.txt` containing 'hello benchflow'"; make verifier/test.sh check exactly that (read `/app/hello.txt`) and write 1.0 to /logs/verifier/reward.txt on success; replace the placeholder assertion in verifier/test_outputs.py (or delete that file); replace the `[REPLACE: ...]` placeholders in the three verifier description files the scaffold also writes — verifier/verifier.md, verifier/rubrics/verifier.md, and verifier/rubrics/verifier.toml (`bench tasks check` rejects any unreplaced placeholder, these three included, so skipping them fails the check below); and make oracle/solve.sh perform the task. Validate: bench tasks check tasks/my-first-task Fix anything it flags (it rejects unreplaced [REPLACE: ...] placeholders). Then run the task with the built-in oracle agent — it executes oracle/solve.sh directly, needs no model or API key, and proves the task + verifier loop: bench eval create \ --tasks-dir tasks/my-first-task \ --agent oracle \ --sandbox docker \ --jobs-dir "jobs/oracle-$(date +%Y%m%d-%H%M%S)" If the console prints "Unknown agent 'oracle' — not in registry ... Will attempt to use as raw command" (older benchflow builds do this), the warning is EXPECTED and the run is still a real oracle run — do not classify it as a broken step. A correct task scores reward 1.0 with the oracle. STEP 8 — Report Summarize for the user: installed version; the eval command used; whether the run was REAL (the three checks); the reward vs. the [PASS]/[FAIL] threshold reading; the artifact paths from step 6; the oracle result for their own task; and suggested next steps (`--concurrency N` for batches, `--skill-mode with-skill` for skill evals, `bench tasks migrate` to convert legacy split-layout tasks to task.md — available in 0.6.0+ — and docs/getting-started.md in the benchflow repo). If any step failed, report exactly which step, the command, and the error — partial honest results beat a fabricated success. ``` # Architecture Source: https://docs.benchflow.ai/architecture # BenchFlow — Architecture *The whole architecture, as one coherent picture — every concept we need, no build-order tiering. Release scoping and milestones are tracked separately (Linear). This document is derived from the sources that count — Han Lee's writing and conversation, our project notes, and the agentic-RL literature — not from the current doc or the current code, both of which are snapshots that follow this, not the other way round.* *** ## What BenchFlow is BenchFlow is the **environment-and-rollout engine for agentic RL** — it turns a stateful environment into evaluated, training-ready trajectory data, for any model and any trainer. **It stops where the gradient starts.** **One engine, three modes.** There is one thing — a *scored rollout*. **Eval** = score it and stop. **Train** = score it and hand the trajectory to a trainer. **Monitor** = score it in production. (Han Lee: *"evaluation, reward and monitoring … it's really all the same thing under different circumstances."*) **The bet.** A complete RL environment is **E = `{T, H, V, S, C}`** — Tasks, Harness, Verifier, **State**, Config (Han Lee, *RL Environments for LLM Agents*). BenchFlow targets the complete E. **State management** — stateful, multi-service environments that can **roll out, roll back, and branch** — is the frontier of agentic RL and the surface BenchFlow is built around. **The boundary.** BenchFlow owns **environment + rollout + reward**. Trainers own **weights + gradients + optimizer**. The **trajectory is the seam** — every RL trainer can consume BenchFlow output without coupling. ## Grounding This architecture rests on three sources, kept honest against each other. **Han's `{T,H,V,S,C}`** (blog, *RL Environments for LLM Agents*) — the environment decomposition. Verbatim: T = "problems the agent tries to solve"; H = the agent harness, "scaffolding that … controls *how* the model interacts, but does not improve what it knows"; V = the verifier, "V: (task prompt, completion, info) → \[0,1]"; S = state, "stateless (fresh starts) … or stateful (persistent across actions/episodes)"; C = configuration, "turn limits, context budgets, sampling temperature, curriculum scheduling." **Han's conversation** (advisory call) — the dynamics the blog's static set does not capture: * *"Environment 总是要 roll out, roll back"* — **roll-out and roll-back are definitional** for an environment. Roll-back = *"snapshot environment and go back to its different stage."* * Branching: an `ask_user`-type interaction *"literally is a checkpoint … to different type of rollout"*, and from it — *"From reward function to a value function … of the current state."* Branching is *"very important for large horizon tasks."* * *"eval = monitoring = reward"* — one activity, observed across **five spaces** (output, action, reasoning, memory, latent). * *"The harness is not meant to be intelligent"* — self-improvement targets the **model and skills**, never the harness; *"skill 是属于 memory"* (skills are memory). * ACP is the mechanism for modelling human interaction inside a rollout. **The agentic-RL literature** — agentic RL is a *"temporally extended, partially observable MDP"* (*The Landscape of Agentic RL*, 2509.02547) — definitionally a branching structure. Tree-structured rollouts give *"step-wise process supervised signals even using only the outcome reward"* and *"more rollouts within a fixed budget of tokens or tool calls"* (*Tree Search for LLM Agent RL* / Tree-GRPO, ICLR 2026). A scan of 13 RL libraries (verifiers, prime-rl, SkyRL, verl, NeMo-RL, Tinker, OpenEnv, Harbor, Terminal-Bench, Inspect, ORS, Gymnasium, agent-lightning) found **all model rollouts linearly** — so a tree-native rollout with environment snapshot/restore is genuine, defensible novelty, and the load-bearing hard part is snapshot/restore of *heavy* environment state (see "The hard part"). ## Design principles 1. **The kernel depends only on contracts.** The call graph is the source of truth; anything exported with no live caller is wired in or deleted. 2. **Four planes, each swappable, each managed + BYO** — Sandbox, Agent, Environment, Reward. 3. **The Rollout is a tree.** An RL episode is a tree of states; a linear rollout is the degenerate degree-1 case. Branch, snapshot, and restore are first-class — they are how a reward function becomes a value function. 4. **Roll-back is definitional.** An environment that cannot snapshot and restore its state is incomplete (Han). `snapshot`/`restore` are real methods, not stubs. 5. **Zero-modification adoption.** A benchmark brings a self-describing package + a manifest; it never subclasses BenchFlow or touches private APIs. 6. **Eval = monitoring = reward** — one activity, scored on the same trajectory, across five spaces. 7. **The environment is a stateful state machine** the framework provisions, snapshots, restores, and tears down. 8. **BenchFlow is the ACP Client** — the "user" is a pluggable policy, not a special actor. 9. **The harness is not intelligent** — its only job is to extract the most from the model; self-improvement targets the model and skills. 10. **Readiness and teardown are framework guarantees** — never the benchmark's burden. 11. **Ship beats design** — a better design that doesn't run loses to an adequate one that does. ## The conceptual model — the planes ``` bench CLI · bf.run() · the environment manifest │ ┌─────────────────▼───────────────────┐ │ KERNEL │ │ Rollout lifecycle · reward · trajectory │ │ depends ONLY on contracts/ │ └──┬──────────┬───────────┬──────────┬──┘ ▼ ▼ ▼ ▼ Sandbox Agent Environment Reward (where) (who) (the world) (how scored) ``` The kernel is **three subsystems** — Rollout lifecycle, reward, trajectory — importing only `contracts/` (four `Protocol`s). Concrete providers (Docker, ACP, `ManifestEnvironment`, `RewardFunc`s) join via a registry. The four planes map onto Han's **E**: | Han's component | BenchFlow | Plane | | ---------------- | -------------------------------------------- | --------------------- | | **T** — Tasks | Task / `task.toml` | kernel concept | | **H** — Harness | the agent + the kernel scaffolding around it | **Agent plane** | | **V** — Verifier | `RewardFunc` / `Rubric` / verifier | **Reward plane** | | **S** — State | the stateful world | **Environment plane** | | **C** — Config | `RolloutConfig` | kernel concept | T and C are kernel concepts (the inputs); H, V, S are the planes that *do* the work; Sandbox is the substrate all three run on. Four planes, one kernel, two kernel-level inputs — that is the whole conceptual surface. ## The execution model — tree-native **A Rollout is one RL episode, and it is a tree.** Han's trajectory is a chain of *state → action → next-state*; a `Branch` makes that chain a tree; classical RL is defined over exactly this tree (a POMDP), and the value function `V(s)` is *defined* as the expected return over the continuations from a state. Modelling the Rollout as a tree is therefore the RL-native choice. The execution model has **three primitives, one derived view, and one authoring form** — all defined on the one tree, not a Russian-doll hierarchy: ``` Job — a set of Rollouts run together (an eval sweep · a GRPO group · a CL sequence) Rollout — one RL episode = a TREE of states (sₜ) PRIMITIVE • Step — one edge of the tree: (reason → act) → (tool-in → tool-out) PRIMITIVE • Branch — the snapshot-and-fork operation; a node with >1 child PRIMITIVE Trajectory — one root-to-leaf path. Computed from the tree, never declared DERIVED VIEW Scene — a declared span carrying a role/skill configuration AUTHORING SUGAR ``` * **The primitives are irreducible.** The tree (`Rollout`), its edges (`Step` — Han's atomic unit; one `Step` is one "turn"), and the `Branch` operation (snapshot + fork). `Branch` is the credit-assignment engine: it evaluates *one state across N continuations* — averaging the children's returns estimates `V(s)`. That is Han's *"from reward function to a value function of the current state"* and Tree-GRPO's peer-reviewed result that a tree yields process supervision from a single outcome reward. A GRPO group run as a shared-prefix tree beats N independent rollouts (more rollouts per token/tool budget). Branches occur at `ask_user`-style interaction checkpoints (one child per option), at GRPO group points, and at value-estimation points. * **`Trajectory` is a derived view** — a pure function of the tree (a path), never declared. It is what serialises out: a linear `prompt / completion / reward / metrics / info` record, the Verifiers/ORS training unit. * **`Scene` is authoring sugar** — the *declaration* form for multi-phase / multi-agent rollouts (`RolloutConfig.scenes`). It desugars completely to per-`Step` role/skill attribution plus config that changes along the tree, and adds no expressive power. It has no runtime object and no lifecycle of its own — `RolloutConfig.scenes` is a desugaring pass that lowers to per-`Step` config. Kept only as a convenient authoring affordance. (The original RFC's instinct — "a phase is just state" — was correct.) **Tree-native is free for the mental model, not for the engine.** A linear rollout genuinely *is* a degree-1 tree, so the *data model* costs nothing extra for the common case. But the *engine* — checkpoint/fork, three-layer snapshot composition, node-addressed scoring, child scheduling — is paid for on day one even by users who only run linear rollouts. That is an accepted cost, not a hidden one: the tree is the correct foundation, and the linear path inherits its machinery. ## Lifecycles Every lifecycle the framework owns, as ordered phases. **Job lifecycle.** `plan` (resolve tasks × agents × repeats) → `schedule` (parallel-independent, or sequential-shared for continual learning) → `run Rollouts` → `aggregate` → `report`. **Rollout lifecycle.** `setup` (resolve config, build the environment object) → `start` (sandbox up) → `provision environment` (Environment plane starts services) → `readiness gate` (framework-guaranteed; the agent never runs before the world is healthy) → `connect agent` (ACP) → `execute` (the tree grows: Steps and Branches) → `verify` (Reward plane scores) → `teardown`. **Branch lifecycle.** `quiesce` (pause the agent at a stable point) → `checkpoint` (snapshot environment state, then container, then agent-session state — in that order, see "The hard part") → `fork` (N children) → `run children` → `score / aggregate` (per-child return → `V(parent)`) → optionally `restore` the winning child's state to continue. **Environment lifecycle** (Han's roll-out / roll-back). `provision` → `readiness` → `query` (expose state to the verifier) → `snapshot` → `restore` → `reset` → `teardown`. `snapshot`/`restore` are definitional — the substrate every `Branch` runs on. **Sandbox lifecycle.** `start` → `exec` / `upload` / `download` / `expose_port` → `snapshot` / `restore` (container-level, coarser than environment-state) → `stop`. A Rollout is checkpointable because three snapshot layers compose — container (Sandbox) ⊃ environment-state (Environment) ⊃ agent-session — but composing them correctly is a real consistency problem (see "The hard part"). The one store that deliberately does **not** roll back with a `Branch` is the continual-learning learner store (capability 5). ## The four planes **Sandbox — where it runs.** Compute substrate. Built-in: `Local` (raw Linux) + `Docker`. Optional: `Daytona`, `Modal`, `Firecracker`, K8s. BYO via the `Sandbox` protocol. Hardening (`lockdown`) is a capability flag. Framework-guaranteed readiness gate + teardown. An environment is declared once and runs on any provider. **Agent — who acts.** The agent under test (eval) or the policy under training — Han's harness, "not intelligent." Protocol: **ACP** (the official `agent-client-protocol`). BYO via `--agent-import-path`. The registry stores agent *declarations* as data, not install code in the kernel. A trainer-served policy endpoint (OpenAI-compatible, hot-swappable) is one agent provider type. The plane's real surface is the `Session` (below) — not just `connect`. ### Skill loading BenchFlow treats mounted skills as agent-native memory, not prompt text. Skills are controlled by one run mode: `no-skill`, `with-skill`, or `self-gen`. `no-skill` hides any task-local `environment/skills` from the agent and strips that directory from copied build contexts. `with-skill` mounts the task's `environment/skills` directory through the selected agent's native skill paths. `self-gen` gives the creator scene only `skill-creator`; the solver scene sees only the generated skills root, never task-bundled skills. Advanced callers can provide a custom `--skills-dir` only in `with-skill` mode. Claude Code reads each skill's frontmatter name and description for native discovery. The full `SKILL.md` body and bundled resources are loaded by the agent when it invokes or reads the skill; BenchFlow does not inline every mounted skill body into the task prompt by default. `BENCHFLOW_SKILL_NUDGE` is an optional prompt nudge layered on top of native discovery. Use `name` to tell the agent which skills are mounted, `description` to include each mounted skill's description, or `full` to include the full `SKILL.md` body. Omit the variable to keep BenchFlow's runtime default off. **Environment — the world (Han's S).** The stateful world the agent acts in. Owns the world's lifecycle: `provision / readiness / query / snapshot / restore / reset / teardown`. See "The Environment plane & the manifest." **Reward — how it's scored (Han's V).** `RewardFunc` / `Rubric` / verifier. `V: (task, completion, info) → [0,1]`, generalised to a graded, multi-space, multi-granularity signal over the trajectory tree. See "Evaluation." ## The four contracts The kernel imports only these. ```python theme={null} class Sandbox(Protocol): # where it runs — container level async def exec(cmd, *, user, timeout) -> ExecResult: ... async def upload(local, remote) -> None: ... async def download(remote, local) -> None: ... async def expose_port(port) -> Endpoint: ... async def snapshot() -> SandboxImage: ... async def restore(image: SandboxImage) -> None: ... async def teardown() -> None: ... class Agent(Protocol): # who acts — Han's harness async def connect(sandbox, role) -> Session: ... def capabilities() -> AgentCapabilities: ... class Session(Protocol): # a LIVE agent session — the Agent plane's real surface async def prompt(text: str) -> StopReason: ... # the task instruction, or a nudge async def cancel() -> None: ... def on_ask_user(handler: AskUserHandler) -> None: ... # agent-initiated; the branch hook @property def steps(self) -> list[Step]: ... # the session's contribution to the tree class Environment(Protocol): # the world — Han's S async def provision(ctx) -> EnvHandle: ... async def readiness() -> ReadinessProbe: ... async def query() -> EnvState: ... # for the verifier async def snapshot() -> StateSnapshot: ... # roll-back: definitional async def restore(snap: StateSnapshot) -> None: ... async def reset() -> None: ... async def teardown() -> None: ... class Reward(Protocol): # how it's scored — Han's V async def score(node: RolloutNode) -> VerifyResult: ... ``` `Session` is part of the contract, not an untyped return — the entire ACP interaction (prompt, nudge, the `ask_user` branch hook) is the Agent plane's seam, so it must be specified to BYO an agent. `Reward.score` takes a `RolloutNode`, and a node **carries its tree context**: `node.path` (root → node), `node.subtree`, `node.state`. One `score` method therefore expresses both outcome reward (read the leaf) and process reward (walk `node.path` across the Action and Reasoning spaces) — there is no per-step-in-isolation scoring. `VerifyResult` = `{reward: float, items: dict[str, float], events: list[RewardEvent], space, granularity}`. ## The Environment plane & the manifest What a benchmark *author* writes is the **manifest** — the entire integration surface. *Write a manifest; your stateful environment runs anywhere and trains anything, with zero framework modification.* The default adapter `ManifestEnvironment` reads it. ```toml theme={null} [environment] name = "chi-bench" image = "chi-bench:latest" # OR base_image + [[services]] (framework-started) owns_lifecycle = true # the image's entrypoint starts the services isolation = "per_task" # OR "persistent" (cross-episode state) [environment.task_selection] mechanism = "env_var" # OR "image" (per-task images, smolclaws-style) key = "CHI_BENCH_TASK_ID" inject_into = "entrypoint" # reaches PID 1, not just exec() [environment.readiness] # the framework gates on this before the agent runs http = ["http://localhost:8023/health"] timeout_sec = 120 [verifier] kind = "agent" hidden_from_agent = ["expectations.json", "tasks/*/fixtures"] ``` **State is a real database**; tools are read-write ops over the schema — which is what makes state snapshot-able, diffable, and verifiable. Two topologies behind one contract: **in-sandbox** (the environment runs in the rollout's own sandbox — the default) and **shared-fleet / sidecar** (a long-lived service fleet + a `TaskDatabase` + `AccountBroker` for multi-tenant per-task accounts — the scale path). **The Stateful Multi-Service Benchmark (SMSB).** ClawsBench and chi-bench are structurally the same machine; the plane hosts both. ClawsBench is the internal dogfood (the manifest's design partner); chi-bench is the external proof — a \~25k-LOC heavy simulator with a thin MCP transport, onboarded via a \~25-line manifest with its environment **untouched**, its \~920 LOC of Harbor coupling collapsing into the manifest. ## Evaluation — the five spaces eval = monitoring = reward. The same scoring runs at train time, at eval time, and in production — only the context differs. A reward signal is read from the trajectory across **five spaces** (Han): | Space | What it checks | | --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- | | **Output** | did it finish the job? (the terminal/verifiable reward) | | **Action** | right actions, no reward-hacking, no out-of-distribution tool use; *did it ask when it should have?* | | **Reasoning** | is the chain-of-thought sound and connected to the action and answer? (CoT monitoring) | | **Memory** | did it update its memory / skills correctly? (diff the store) | | **Latent** *(future)* | with interpretability access — SAEs over post-attention embeddings. No benchmark needs it yet; named so it isn't reinvented later, not built. | Every reward record is tagged **`(space, granularity, value)`**. Granularity is **terminal** (the whole trajectory) or **step** (one edge) — an episode-level scalar alone is inadequate beyond \~50 steps; the tree's structure supplies finer credit. **Process reward** is read by walking a node's `path` across the Action and Reasoning spaces — *not* by scoring each step in isolation (process supervision "hard to judge" per-step — Han). The wire formats `reward.txt` / `reward.json` cross the sandbox boundary; the in-kernel model is `VerifyResult` + `RewardEvent`. ## The interaction model — ACP Human interaction is modelled through ACP's role split: **BenchFlow is the ACP Client; the "user" is a pluggable User Model inside the Client role.** Two channels carry everything: * `session/prompt` (Client → Agent) — the task instruction and every **nudge** (user-initiated follow-up). * `request_permission` / `ask_user` (Agent → Client, with enumerated options) — agent-initiated, surfaced through `Session.on_ask_user`. `ask_user` with enumerated options is the **branchable interaction primitive** — finite options ⇒ a finite, scoreable interaction tree (each option is one `Branch` child). The interaction tool is never hard-coded as "step one"; the agent chooses to use it, and the **Action space** scores *whether it asked* — an under-specified task makes "ask the user" the correct behaviour, and failing to ask is a negative reward (Han). User Model modes: scripted / simulated (LLM persona) / real-human / auto. (Branching is not a User Model mode — it is a property of the `Rollout` tree.) ## The edges — adapters & trainers The manifest is BenchFlow's native format; **adapters translate every other format to it.** **Inbound env adapters** — Harbor, Inspect, ORS, PrimeIntellect/Verifiers environments → run foreign benchmarks natively. **Terminal-Bench tasks run through the Harbor adapter** (Harbor is itself terminal-bench-derived). **Outbound — the trainer seam.** A scored trajectory exports as a **Verifiers / ORS JSONL record** (`prompt / completion / reward / metrics / info`). Being a Verifiers/ORS-compatible producer yields a trainer — prime-rl — with zero trainer code. BenchFlow is a rollout *service*; trainers (Tinker, verl, NeMo-RL) stay external. The trajectory is the seam. ## How a Task flows through the architecture A **Task** (Han's T) is the problem spec — `task.toml` + instruction + the environment package + the verifier. It is a kernel concept, and it is what wires the planes together for one run: ``` Task ─┬─→ selects the Environment package + manifest ───→ Environment plane provisions S ├─→ carries the instruction / prompt ───→ Agent plane (H) receives it ├─→ names the verifier + hidden fixtures ───→ Reward plane (V) scores └─→ carries config (turn limits, budgets) ───→ RolloutConfig (C) │ ▼ one Rollout (a tree) runs in a Sandbox │ ▼ Trajectory(s) + reward ───→ export ───→ trainer ``` One Task → one Rollout tree → one or more Trajectories. A Job is many Tasks (or one Task × many repeats). `{T,H,V,S,C}` is not an abstraction layered on top — it *is* the wiring diagram of a single run. ## The eight capabilities — how each fits The architecture is one shape; these are the eight things it must carry. Capabilities 1–6 and 8 are benchmark-forced — "done" = that benchmark runs clean. Capability 7 is the substrate the others ride on, not a benchmark. | # | Capability | How it fits the architecture | | - | ------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | 1 | **SkillsBench** | An Environment-plane benchmark package (skills + skill-eval tasks). Skills are **memory** (Han); the Reward plane's **Memory space** scores skill use and skill updates. Skills are installed as per-`Step` config (the `Scene` desugaring) and deployed into the sandbox. | | 2 | **ClawsBench** | The SMSB on the Environment plane — `base_image` + `[[services]]`, framework-started, `image` task-selection. The internal dogfood; the manifest's design partner. | | 3 | **chi-bench** | The same SMSB archetype — `image` + `owns_lifecycle = true` + `env_var` task-selection. The external proof: onboarded by a \~25-line manifest, environment untouched. | | 4 | **followupbench (NudgeBench)** | The **ACP interaction model** (`session/prompt` nudges + `ask_user` via `Session.on_ask_user`) + the **tree-native Rollout** — every interaction checkpoint is a `Branch` — + the **Action space** reward scoring *whether the agent followed up / asked*. | | 5 | **Continual learning** | A **Job run in `sequential-shared` mode**: Rollouts run in order over a persistent **learner store** (memory + skills). The store is versioned (a generation stamped per rollout) and rollback-capable; the **Memory space** tracks improvement, drift, and adoption. Skills stay useful *only if continuously evolved* (Han). The learner store is the one snapshot layer that does not roll back with a `Branch`. | | 6 | **RL-native** | The whole execution model: the Rollout is a tree, the Trajectory is a path, the Reward contract scores any node, and the trajectory exports as a trainer-ready record. Agentic RL is a temporally-extended POMDP — and the architecture is shaped like one. | | 7 | **Branching, rollback, Han's framework** | *Not a benchmark — the RL-native substrate itself.* First-class `Branch`; `Environment.snapshot`/`restore` as definitional roll-back; the value-function purpose of the tree; the five spaces; eval = monitoring = reward; the non-intelligent harness. Capabilities 4–6 ride on it. | | 8 | **Env adapters — Harbor / PrimeIntellect / OpenReward** | The **edges**: inbound adapters translate foreign formats to the manifest; **Terminal-Bench backward compatibility** rides the Harbor adapter; outbound, the trajectory exports to Verifiers/ORS. | All eight land on one architecture — four planes, a tree-native Rollout, an adapter edge. None requires a new top-level concept. ## The hard part — honest risk The library scan is unambiguous: **no agentic-RL library ships environment snapshot/restore.** Tree-GRPO branches a *token prefix*; Inspect's `fork()` deep-copies *conversation* state, not the sandbox; its checkpoint system is resume-only — its design note says *"reality doesn't have a fork command."* BenchFlow's bet is to branch a **heavy stateful environment** — a mock-Gmail SQLite database, a healthcare simulator, eventually a K8s cluster. The `Branch` checkpoint is genuinely three unsolved problems, not one: 1. **Environment-state snapshot** — DB dump/restore, copy-on-write volumes, fork-able service processes. Designed deliberately, environment-class by environment-class — not one generic call. 2. **Agent-session snapshot** — freezing and restoring a *live ACP session* (and the running agent process and its context behind it) is the same class of hard problem. Inspect can only deep-copy conversation state precisely because it cannot snapshot the process. The architecture does not get this for free. 3. **Cross-layer consistency** — the three snapshot layers (container / environment-state / agent-session) have different consistency models; a naïve capture can produce a container snapshot and a DB snapshot that disagree about a write in flight. The `Branch` lifecycle therefore **quiesces the agent first**, then snapshots environment → container → session in order. Tree-native rollout *structure* is proven and safe to commit to. The `Branch` *checkpoint* — all three layers — is the frontier: it is where the engineering risk concentrates and where the moat is, and it must be designed deliberately, not hand-waved as one call. ## Adaptations — the decision log So decisions are not re-litigated. | Adaptation | From → To | Why | | --------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | **One picture, no build-order tiering** | a core/deferred two-altitude split → the whole architecture as one coherent overview | The overview describes *what we need*; sequencing is a roadmap concern (Linear), not a property of the concepts. | | **The Rollout is a tree** | a linear Rollout with optional, deferred branching → a tree-native Rollout; linear = degree-1 | Agentic RL is a POMDP; `V(s)` is defined over a tree; Tree-GRPO shows the tree manufactures credit assignment. Branching is the engine, not a feature. | | **`snapshot`/`restore` are real** | platform-layer `NotImplementedError` stubs → definitional methods on the Environment contract | Han: *"Environment 总是要 roll out, roll back."* | | **Branching's purpose is the value function** | a user-feedback feature → credit assignment / `V(s)` estimation; `ask_user` and GRPO groups are two cases of it | Han: *"from reward function to a value function."* Tree-GRPO confirms it. | | **`Session` is in the contract** | the Agent plane returned an untyped `Session` → `Session` is a specified `Protocol` | The whole ACP interaction is the Agent seam; a shallow `connect`-only contract can't carry BYO agents. | | **Scene fully desugars** | a runtime object with its own lifecycle → pure authoring sugar; `RolloutConfig.scenes` lowers to per-`Step` config | Scene adds no expressive power; a phase is just state (the RFC's original instinct). | | **Renamed `Batch` → `Job`** | `Batch` for a set of Rollouts → `Job` | "Batch" means the gradient minibatch in every trainer — a collision at the trainer seam. | | **Manifest as the only seam** | benchmarks subclass framework internals → a declarative manifest, zero framework modification | A benchmark must never modify the framework. | | **eval = monitoring = reward** | three framings → one engine, three modes | Han's single biggest complexity reducer. | ## Appendix — research validation Checked against the recent agentic-RL literature; the field's shape matches. * Agentic RL = a temporally-extended POMDP — definitionally a branching structure. *(The Landscape of Agentic RL, 2509.02547)* * Tree-structured rollouts yield process supervision from a single outcome reward and more rollouts per token/tool budget. *(Tree Search for LLM Agent RL / Tree-GRPO, ICLR 2026, 2509.21240)* * All 13 surveyed RL libraries model rollouts linearly with no environment snapshot/restore — tree-native + heavy-environment snapshot is real novelty, not a reinvention. * Rollout-as-a-service decoupled from training; the trajectory is the seam. *(ProRL Agent; PrimeIntellect Environments Hub)* * Continual learning = a base policy + a persistent, evolving skill library; version and roll back the store. *(MetaClaw, MemSkill, SkillLearnBench)* **Verdict:** the architecture is consensus-correct on shape and deliberately ahead of the field on one primitive — the `Branch` checkpoint (environment + agent-session + container snapshot) for stateful branching. The risk is execution of that primitive, not the design. # Benchmark adoption Source: https://docs.benchflow.ai/benchmark-adoption # Benchmark adoption Adopt an upstream benchmark into a BenchFlow benchmark with `bench eval adopt`. ## What the router is `bench eval adopt` is the benchmark-adoption router. It *routes* an external benchmark into `benchmarks//` — scaffold, codex-driven conversion, and a parity gate — so the result is a first-class BenchFlow benchmark. It sits upstream of evaluation: the router *adopts*, while `bench eval create` *runs* the resulting tasks. Once `bench eval adopt verify ` reports `parity-confirmed`, you point `bench eval create` at the converted tasks and run them like any other benchmark. (These commands were `bench agent create|run|verify` before 0.6; the old names still work as deprecated aliases through 0.6 and are removed in 0.7.) Three subcommands form the adopt → verify loop: ``` $ bench eval adopt --help ╭─ Commands ───────────────────────────────────────────────────────────────────╮ │ init Scaffold benchmarks// for a new benchmark adoption. │ │ convert Drive the CONVERT.md workflow by launching the host codex CLI. │ │ verify Run the parity gate for an adopted benchmark; emit a verdict. │ ╰──────────────────────────────────────────────────────────────────────────────╯ ``` The reference for what a finished adoption looks like is [`benchmarks/programbench/`](../benchmarks/programbench/); the conversion contract is [`benchmarks/CONVERT.md`](../benchmarks/CONVERT.md). The router embeds both into the conversion workflow for you. ## `bench eval adopt init ` `init` writes a deterministic scaffold under `benchmarks//`, matching the reference layout and the CONVERT.md contract. Use `--benchmarks-dir` to target a directory other than the repo's `benchmarks/`: ``` $ bench eval adopt init webarena-lite --benchmarks-dir /tmp/router-docs/benchmarks Scaffolded /tmp/router-docs/benchmarks/webarena-lite README.md __init__.py benchflow.py benchmark.yaml main.py parity_experiment.json parity_test.py run_webarena_lite.py webarena-lite.yaml ``` That produces this tree: ``` webarena-lite/ ├── __init__.py ├── benchflow.py # converter: source instances → Harbor task dirs ├── main.py # converter CLI delegator ├── parity_test.py # structural / eval / side-by-side parity checks ├── parity_experiment.json # recorded parity results (read by verify) ├── benchmark.yaml # standard benchmark descriptor ├── run_webarena_lite.py # runner: convert, then evaluate via BenchFlow ├── webarena-lite.yaml # BenchFlow job config (how to run) └── README.md # generated workflow notes ``` What each file is for: * **`benchflow.py`** — the converter. Its documented `convert()` / `convert_all()` entry points are `NotImplementedError` stubs that point at CONVERT.md step 2; you fill them in to map each source instance to a Harbor-format task directory (`task.toml`, `instruction.md`, `environment/Dockerfile`, `tests/test.sh`). * **`parity_test.py`** — the parity harness, with `--mode full | eval-parity | side-by-side` (CONVERT.md steps 3–5). Side-by-side parity records the per-criterion `original_verdict` / `adapted_verdict` pairs that `verify` scores. * **`parity_experiment.json`** — the recorded parity results `verify` reads. The scaffold writes a `status: "template"` placeholder with empty `conversion_parity.tasks` and `reward_distribution_parity.samples`; you populate it from a real parity run. * **`benchmark.yaml`** — the standard descriptor (name, conversion method, verification method, parity tallies). Fields start as `TODO`/`0`. `main.py`, `run_webarena_lite.py`, and `webarena-lite.yaml` are the converter CLI delegator, the convert-then-evaluate runner, and the BenchFlow job config respectively. ### Fail-closed behavior `init` refuses to overwrite an existing benchmark — re-running it is an error, not a silent clobber: ``` $ bench eval adopt init webarena-lite --benchmarks-dir /tmp/router-docs/benchmarks benchmark already exists: /tmp/router-docs/benchmarks/webarena-lite (refusing to overwrite) ``` Names must be lowercase slugs (leading letter, single internal hyphens). The slug is also the security floor — it keeps `init`/`verify` from being steered outside `benchmarks/`. An uppercase or underscored name is rejected: ``` $ bench eval adopt init WebArena_Lite --benchmarks-dir /tmp/router-docs/benchmarks invalid benchmark name 'WebArena_Lite': use a lowercase slug like 'my-bench' (letters/digits, single internal hyphens, leading letter) ``` Both fail-closed cases exit non-zero. ## `bench eval adopt convert [--name]` `convert` drives the conversion. It assembles an adoption prompt — the source, the target `benchmarks//` path, the adoption skills (CONVERT.md, the programbench worked example, the parity harness), and the full embedded CONVERT.md guide — then launches the host `codex` CLI to do the conversion toward a pull request. If you omit `--name`, the slug is derived from the source basename (so `.../webarena` becomes `webarena`). Use `--dry-run` to print the exact command the router would launch without running it: ``` $ bench eval adopt convert https://github.com/web-arena-x/webarena --name webarena-lite --dry-run codex exec --cd /path/to/benchflow --skip-git-repo-check --sandbox workspace-write '# Benchmark adoption: webarena-lite Adopt the source benchmark below into a BenchFlow benchmark by following the conversion guide. Produce the converter, parity tests, metadata, and task directories, then open a pull request. Source benchmark: https://github.com/web-arena-x/webarena Target directory: benchmarks/webarena-lite/ ## Adoption skills - conversion-guide: benchmarks/CONVERT.md - reference-benchmark: benchmarks/programbench/ (worked example) - parity-harness: parity_test.py + parity_experiment.json (verify gate) ## Conversion guide (benchmarks/CONVERT.md) # Benchmark Conversion Guide ... ## Definition of done - benchmarks/webarena-lite/ has benchflow.py, parity_test.py, parity_experiment.json, benchmark.yaml, run_webarena_lite.py, README.md - `bench eval adopt verify webarena-lite` reports parity-confirmed' ``` The full prompt embeds CONVERT.md verbatim (elided above). The `codex exec` argv is constructed deterministically: it runs in the repo root (`--cd `), with `--skip-git-repo-check` and `--sandbox workspace-write`. Pass `--model` to set the codex driver model and `--codex-bin` to point at a different codex binary. A live run (drop `--dry-run`) requires codex credentials and fails closed without them — set `OPENAI_API_KEY` (or `CODEX_API_KEY`), or run `codex login` to create `~/.codex/auth.json`. Without credentials `convert` errors before assembling any context: ``` codex needs credentials to launch: set OPENAI_API_KEY (or CODEX_API_KEY), or run `codex login` to create ~/.codex/auth.json ``` The codex run is the manual-validation step — it iterates on the converter and parity tests until `bench eval adopt verify` confirms parity. ## `bench eval adopt verify ` `verify` is the gate that closes the loop. It reads the adopted benchmark's `parity_experiment.json` and emits a confidence verdict. The gate is *parity only*: a faithful conversion must reproduce the original's behavior on identical inputs — including any reward-hackability the original has. It never "improves" or sanitizes the source. It scores two layers: * **Conversion parity (deterministic floor)** — every compared criterion's converted verdict must match the original's verdict on identical inputs. * **Reward-distribution parity (statistical layer)** — every legacy-vs-converted reward delta must sit within `--tolerance` (default `0.02`). A layer with no recorded data does not block the verdict. The three verdicts: | Verdict | Meaning | | ----------------------- | ------------------------------------------------------------------------------- | | `parity-confirmed` | Every recorded layer agrees; high-confidence the conversion is faithful. | | `parity-divergent` | A criterion disagrees or a reward delta exceeds tolerance. | | `insufficient-evidence` | No recorded comparisons at all — run `parity_test.py` and record results first. | A freshly scaffolded benchmark has no recorded parity, so it is `insufficient-evidence` and exits non-zero: ``` $ bench eval adopt verify webarena-lite --benchmarks-dir /tmp/router-docs/benchmarks Verdict: insufficient-evidence conversion: 0/0 criteria agree (rate 0.0000) Insufficient evidence: no recorded parity comparisons. Run parity_test.py and record results before trusting the conversion. ... ``` ### A parity-confirmed run Populate `parity_experiment.json` from a parity run. `verify` reads per-criterion verdicts under `conversion_parity.tasks` and reward samples under `reward_distribution_parity.samples`: ```json theme={null} { "experiment": "side-by-side-parity", "benchmark": "webarena-lite", "status": "recorded", "judge_model": "gemini-3.1-flash-lite", "conversion_parity": { "tasks": [ { "task_id": "shopping-001", "n_criteria": 2, "criteria_results": [ {"criterion_id": "C-001", "original_verdict": "pass", "adapted_verdict": "pass", "agreement": true}, {"criterion_id": "C-002", "original_verdict": "fail", "adapted_verdict": "fail", "agreement": true} ] }, { "task_id": "reddit-002", "n_criteria": 1, "criteria_results": [ {"criterion_id": "C-001", "original_verdict": "pass", "adapted_verdict": "pass", "agreement": true} ] } ] }, "reward_distribution_parity": { "samples": [ {"task_id": "shopping-001", "legacy_reward": 0.50, "converted_reward": 0.50}, {"task_id": "reddit-002", "legacy_reward": 1.00, "converted_reward": 1.00} ] } } ``` With every criterion agreeing and every reward delta at zero, the verdict is `parity-confirmed` and `verify` exits zero: ``` $ bench eval adopt verify webarena-lite --benchmarks-dir /tmp/router-docs/benchmarks Verdict: parity-confirmed conversion: 3/3 criteria agree (rate 1.0000) reward: max abs delta 0.0000 (tolerance 0.0200) High-confidence: the converted evaluation reproduces the original's verdicts on every compared criterion and stays within reward tolerance. ``` ### A parity-divergent run Flip one criterion so the converted verdict no longer matches the original (here `C-002`'s `adapted_verdict` goes from `fail` to `pass`). The deterministic floor trips, the verdict becomes `parity-divergent`, and `verify` prints a draft GitHub issue body for the support path: ``` $ bench eval adopt verify webarena-lite --benchmarks-dir /tmp/router-docs/benchmarks Verdict: parity-divergent conversion: 2/3 criteria agree (rate 0.6667) reward: max abs delta 0.0000 (tolerance 0.0200) Divergence found: the conversion does not yet reproduce the original's behavior — iterate, then open an issue for support. ## Benchmark adoption parity: webarena-lite **Verdict:** parity-divergent Divergence found: the conversion does not yet reproduce the original's behavior — iterate, then open an issue for support. ### Conversion parity (deterministic floor) - criteria compared: 3 - agreed: 2 - agreement rate: 0.6667 - shopping-001/C-002: original=fail converted=pass ### Reward-distribution parity (statistical layer) - samples: 2 - max abs delta: 0.0000 - tolerance: 0.0200 ### Ask Parity could not be closed for this conversion. The translation must reproduce the original's behavior on identical inputs (including any reward-hackability it has). This draft has NOT been filed — review it, iterate on the converter, and open it manually if you need support. ``` The draft is **never filed automatically** — it is printed for a human to review and open if they need support. Pass `--issue-out PATH` to write it to a file instead of stdout: ``` $ bench eval adopt verify webarena-lite --benchmarks-dir /tmp/router-docs/benchmarks --issue-out /tmp/router-docs/divergence.md Verdict: parity-divergent ... Issue draft written to /tmp/router-docs/divergence.md ``` ### The `--roundtrip-task` structural hook By default `verify` scores the recorded `parity_experiment.json` at the benchmark level. Pass `--roundtrip-task ` to also run the structural round-trip conformance check on one concrete task tree (it reuses the existing Harbor round-trip parity utility). It is opt-in because that harness needs a concrete task directory, which the benchmark-level verdict does not require. `verify` exits non-zero for `parity-divergent` and `insufficient-evidence`, and errors if the benchmark was never adopted: ``` $ bench eval adopt verify nonexistent-bench --benchmarks-dir /tmp/router-docs/benchmarks benchmark not adopted: /tmp/router-docs/benchmarks/nonexistent-bench — run `bench eval adopt init nonexistent-bench` first ``` ## From adoption to evaluation Once `verify` reports `parity-confirmed`, the benchmark is a normal BenchFlow benchmark: run its tasks with `bench eval create` (see [Running benchmarks](./running-benchmarks.md)), using the job config the scaffold generated. The router's job ends at `parity-confirmed`; evaluation takes it from there. # Concepts Source: https://docs.benchflow.ai/concepts # Concepts The mental model for benchflow. Read once, then refer back from the how-tos. *** ## The five primitives | Primitive | What it is | | --------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Task** | A directory on disk: a `task.md` document (YAML frontmatter + prompt body) plus `environment/Dockerfile` for the sandbox, `verifier/` checks, and optional `oracle/` — or the legacy split layout (`task.toml` + `instruction.md` + `tests/` + `solution/`). Authored once, evaluated many times. | | **Agent** | A registered ACP-speaking program (Claude Code, Gemini CLI, OpenCode, etc.). Identified by name (`"gemini"`, `"opencode"`) plus an optional model ID. Use the `acpx/` prefix (e.g. `acpx/gemini`) to route through [ACPX](https://acpx.sh/), a headless ACP client with persistent sessions and crash recovery. | | **Environment** | The sandbox where the agent runs and the verifier checks the result. Docker locally, Daytona for cloud, Modal for serverless/GPU. Abstracted behind the `Sandbox` protocol — bring your own sandbox backend. | | **Verifier** | The test runner that scores the rollout. Its entry point is a `test.sh` script (native `verifier/test.sh`, legacy `tests/test.sh`) — which typically runs `pytest` against the workspace the agent left behind. For subjective tasks, use an [LLM-as-judge](./llm-judge.md) verifier scored against a rubric. Outputs `rewards: {reward: float}`. See the [verifier file map](#verifier-file-map) for which file lives where in native vs legacy packages. | | **Rollout** | One agent run on one task. Holds the lifecycle (setup → start → install → execute → verify → cleanup). All higher-level primitives below are built on Rollouts. | *** ## Rollout lifecycle A `Rollout` is decomposable: each phase is a callable method, you can either run them in sequence or invoke `Rollout.run()` to execute all six in order. Multi-agent flows reuse phases (e.g. `connect` + `execute` + `disconnect` repeats per role). ``` ┌──────────────────────────────────────────────────────────────┐ │ Rollout.run() │ │ │ │ setup() resolve config, create sandbox env handle │ │ ↓ │ │ start() start container, upload task files │ │ ↓ │ │ install_agent() install agent binary, write credentials, │ │ set up sandbox user │ │ ↓ │ │ ┌─ connect_as(role) ◄─── multi-agent loops here │ │ │ execute(prompts) each role's turn │ │ └─ disconnect() │ │ ↓ │ │ verify() harden sandbox, run pytest, score │ │ ↓ │ │ cleanup() kill agent procs, stop container │ └──────────────────────────────────────────────────────────────┘ ``` Each phase has a name, a clear contract, and is independently testable. `Rollout.run()` is the convenience that calls them in order. ```python theme={null} import benchflow as bf from benchflow import RolloutConfig, Scene from pathlib import Path config = RolloutConfig( task_path=Path("tasks/edit-pdf"), scenes=[Scene.single(agent="gemini", model="gemini-3.1-pro-preview")], environment="daytona", ) result = await bf.run(config) # full lifecycle print(result.rewards) # {'reward': 1.0} ``` *** ## Scenes, Roles, Turns A **Scene** is authoring sugar for Step metadata. Inside a Scene: * **Roles** are the agents that participate (one or more). * **Turns** are the prompt sequence — which Role acts when, and what they're told. * All Roles share the same sandbox filesystem. Before rollout execution, BenchFlow desugars Scenes into explicit rollout Steps carrying role, prompt, and skill attribution. Scene has no runtime object, scheduler, message router, or lifecycle. ```python theme={null} Scene( name="review-loop", roles=[ Role(name="coder", agent="opencode", model="anthropic/claude-sonnet-4-6"), Role(name="reviewer", agent="gemini", model="gemini-3.1-pro-preview"), ], turns=[ Turn(role="coder"), Turn(role="reviewer", prompt="Review the current workspace."), Turn(role="coder", prompt="Read the reviewer's feedback and revise."), ], ) ``` A Rollout may have multiple Scenes — used for staged flows like "skill generation → solve" (BYOS / Bring Your Own Skill). Same sandbox, sequential Scenes. *** ## The User abstraction (multi-round, single-agent) Sometimes you want the agent to take multiple turns guided not by another LLM but by a Python callback that watches what happened and decides what to say next. That's a **User**. A User is a `BaseUser` subclass (or `FunctionUser` wrapping a function) with two methods: * `setup(instruction, solution)` — once, before round 0 * `run(round, instruction, round_result) → str | None` — per round; return `None` to stop the loop Between rounds, BenchFlow executes `soft_verify()` (verifier without the destructive parts of full hardening), gives the user the round's `RoundResult` (trajectory, rewards, verifier output, tool count), and lets the user decide round N+1's prompt. Use `BaseUser` when the loop logic is rule-based (compress instruction → show test failures as hints → stop on pass). See [`progressive-disclosure.md`](./progressive-disclosure.md) for the full guide. *** ## Verifier, sandbox, hardening Once the agent stops, the verifier runs. Its entry point is the task's `test.sh` script — uploaded to `/verifier` for native packages (`/tests` for legacy ones) — executed against the workspace the agent left behind. benchflow runs `test.sh` **as a script** (it `chmod +x`'s the file and executes it directly; a native `script` strategy runs `cd /verifier && `). It never hands `test.sh` to `pytest` — pytest cannot collect a shell script as a test target. Most `test.sh` scripts *invoke* pytest internally. For those invocations, benchflow applies hardening through `PYTEST_ADDOPTS` in the verifier environment — every pytest run inside `test.sh` inherits roughly: ```text theme={null} PYTEST_ADDOPTS="-c /dev/null --confcutdir= --rootdir= -p no:cacheprovider" ``` where `` is `/verifier` for native packages (`/tests` for legacy), and `` is the agent workspace (`/app` for Harbor/SWE-bench conventions, `/root` for SkillsBench — injected dynamically). `-c /dev/null` blocks `pyproject.toml`/`pytest.ini` discovery and `--confcutdir` blocks `conftest.py` walk-up beyond the verifier dir. Tasks that do not use pytest (e.g. a `test.sh` that diffs files and writes `reward.txt` directly) are scored the same way — pytest is just the most common tool, not a requirement. Between agent and verifier, benchflow **hardens** the sandbox to prevent the agent from gaming the score: * Kill any lingering agent processes * Restore build-config files (setup.py, pyproject.toml, …) to their pre-agent snapshots * Delete agent-injected `conftest.py`, `sitecustomize.py`, `.pth` files * Lock the workspace to root, set restrictive PYTHONPATH/PATH for the verifier process * Run pytest with plugin auto-discovery off, only allowing plugins declared in the task config (`[verifier] pytest_plugins` in `task.toml`, or auto-discovered root-owned plugins) This catches the BenchJack and Meerkat exploit families documented in the historical (0.2.x-era) labs [`docs/labs/benchjack-sandbox-hardening/`](../docs/labs/benchjack-sandbox-hardening/) and [`docs/labs/reward-hack-matrix/`](../docs/labs/reward-hack-matrix/). When a task ships a legitimate `conftest.py` (e.g. qutebrowser uses one to break a real circular import), the task opts out via `task.toml`: ```toml theme={null} [verifier.hardening] cleanup_conftests = false ``` See [`progressive-disclosure.md`](./progressive-disclosure.md#per-task-hardening-opt-outs) for the full opt-out list. ### Verifier file map Native `task.md` packages and the legacy split layout name their verifier files differently. The runtime resolves native files first and falls back to the legacy names, so a task ships **one** of each row, not both: | What it is | Native (`task.md`) package | Legacy split layout | Sandbox path | | -------------------------------------- | ----------------------------------------------------------------- | ---------------------------------------------------------- | ---------------------------------------------------------------- | | Verifier directory | `verifier/` | `tests/` | `/verifier` (native), `/tests` (legacy) | | Script entry point | `verifier/test.sh` | `tests/test.sh` | executed as a script (chmod +x then run) inside the verifier dir | | Strategy declaration (how it's scored) | `verifier/verifier.md` | — (legacy uses `[verifier]` in `task.toml`) | not uploaded as a runtime target; selects the strategy | | LLM-judge rubric | `verifier/rubrics/verifier.md` + `verifier/rubrics/verifier.toml` | `tests/rubric.toml` (also `rubric.json`, Harvey-LAB style) | downloaded for the judge | A plain `test.sh` is a complete verifier on its own: with no `verifier.md` strategy declared, the runtime just executes it. `verifier/verifier.md` declares *how* a task is scored (script / llm-judge / reward-kit / agent-judge / ors-episode) and is the native equivalent of the legacy `[verifier]` section in `task.toml`. The native LLM-judge rubric lives under `verifier/rubrics/` (both a human-readable `verifier.md` and a machine-readable `verifier.toml`), not in a single top-level `rubric.toml`. For the native verifier document and its strategy table see [Native task.md authoring](./task-authoring-task-md.md); for the legacy `[verifier.judge]` rubric path see [LLM-as-judge](./llm-judge.md). *** ## Multi-turn vs multi-round vs multi-scene Three different axes — easy to confuse, worth pinning down: | Axis | What changes | Example | | --------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- | | **Multi-turn** | Same Role, multiple prompts within one Scene. The ACP session persists; the agent has continuous memory. | One coder gets prompted twice: "fix the bug", then "now write a test". | | **Multi-round** | Same Role, multiple `connect → execute → disconnect` cycles. New ACP session each round; sandbox state persists; a Python `User` callback decides each round's prompt. | Progressive disclosure on SWE-bench Pro: round 0 terse spec, round 1 hints with failing tests, round 2 full spec. | | **Multi-scene** | Multiple Scenes in one Rollout. Sandbox state persists; agent process and ACP session restart between Scenes. | BYOS: Scene 1 generates a skill, Scene 2 solves the task using it. | Single-agent simple runs use none of these. Pick the axis based on what state needs to persist (memory? sandbox? both?). *** ## Trajectories and rewards Every agent action is captured as an event in the **trajectory** — tool calls, agent messages, agent thoughts. A `RolloutResult` (aliased as `RunResult`) has the full trajectory plus tool count, plus rewards from the verifier and any error. `rewards` is a dict produced by the task's verifier. Convention: `{"reward": float}` where 1.0 = pass, 0.0 = fail. Tasks may add additional metrics (e.g. `exact_match`, `partial_credit`). Trajectories are written to `///trajectory/acp_trajectory.jsonl` (the `--jobs-dir` directory, default `jobs/`). Use them for replay, debugging, or training data. *** ## Where to go next * [Getting started](./getting-started.md) — install, run your first eval. * [Task authoring (native task.md)](./task-authoring-task-md.md) — write a task as a single `task.md` document plus `environment/` and `verifier/`. * [Task authoring (legacy split layout)](./task-authoring.md) — write a task with `task.toml` + `tests/` + `solution/`. * [LLM-as-judge](./llm-judge.md) — use an LLM to score subjective tasks against a rubric (see the [verifier file map](#verifier-file-map) for native vs legacy rubric paths). * [Progressive disclosure](./progressive-disclosure.md) — the User abstraction; SWE-bench Pro case study. * [Use cases](./use-cases.md) — multi-agent patterns (coder/reviewer, simulated user, BYOS, stateful environments). * [CLI reference](./reference/cli.md), [Python API reference](./reference/python-api.md). * [Skill evaluation](./skill-eval.md) — when the artifact is a skill, not a workspace. # Continue runs Source: https://docs.benchflow.ai/continue-runs # Continuing timed-out runs (`benchflow continue`) `benchflow continue` resumes a previous, **unfinished** (timed-out) agent run to completion. It is a standalone tool — it does **not** modify benchflow's normal `eval`/run path — and currently targets the **`openhands`** agent. The goal is a *transparent* resume: the continued run behaves as if the original timeout had simply been larger. The agent keeps its exact context and environment and continues its own loop with **no injected prompt**. ## The problem it solves A finished run keeps nothing of the container — cleanup tears the sandbox down. What survives on disk is the run folder: `config.json`, `result.json`, `prompts.json`, and `trajectory/llm_trajectory.jsonl`. So a historical timeout has only its *trajectory* + the *task*; there is no saved container to restore. `benchflow continue` reconstructs the missing state from the trajectory. ## How it works — record-replay The recorded `llm_trajectory.jsonl` is the exact sequence of LLM request/response pairs from the original run. `benchflow continue`: 1. **Loads** the original run folder and the recorded exchanges. 2. **Boots a fresh, pristine sandbox** from the same base image. 3. Stands up a **replay proxy** that OpenHands talks to via `LLM_BASE_URL`. For the first *N* requests it returns the recorded responses **in order**, so the agent re-executes its own past decisions *for real* — rebuilding the byte-exact workspace and its exact internal conversation/event state. 4. When the recorded responses run out (the timeout cut-point), the proxy flips to the **live model** and the agent continues — no new prompt. 5. **Re-verifies** with the task verifier and writes a new HF-compatible folder, with a stitched `llm_trajectory.jsonl` (recorded prefix + live suffix) and `continued_from` provenance — a drop-in replacement for the timed-out entry. Because the agent rebuilds its own state by re-doing its own steps, no reverse-engineering of OpenHands internals is needed, and the result is a single continuous run rather than a fresh agent on a warm filesystem. ## Usage ```bash theme={null} benchflow continue path/to/original/run-folder \ --tasks-dir path/to/tasks # where the task source (verifier) lives ``` The uploaded run folder does **not** ship the task's verifier, so point `--tasks-dir` at the directory containing the task (matched by name). If the `task_path` recorded in `config.json` still exists on disk, `--tasks-dir` is optional. ### Options | Flag | Default | Meaning | | --------------------- | ------------------------- | -------------------------------------------------------------------- | | `--tasks-dir DIR` | recorded `task_path` | Task source (instruction + verifier). | | `--model MODEL` | original run's model | Override the **live-continuation** model. | | `--timeout SEC` | original run's timeout | Wall-clock budget for the continuation. | | `--output DIR` | `/continued` | Output jobs dir for the new run. | | `--require-timeout` | off | Refuse runs whose recorded status isn't a timeout. | | `--strict-divergence` | off | Abort if replay leaves the original rails. | | `--replay-only` | off | Rebuild via replay and stop at the cut-point (no live model needed). | ### Models and credentials * The **live-continuation model** defaults to the original run's model so the continuation is a faithful continuation of the same brain. Tests use `--model gemini-3.1-flash-lite-preview` for a cheap path. * The **replay phase needs no API key** — responses are served from the recording. Only the **live continuation** calls the real provider, so the host needs that provider's credentials (e.g. `GEMINI_API_KEY`) in its environment. `--replay-only` skips the live leg entirely. ## Limitations and caveats * **`openhands` only** for now (the proxy seam relies on `LLM_BASE_URL`). * **Replay fidelity is best-effort.** Replay re-runs the original shell commands for real; if a command's output diverges from the original (network, timestamps, nondeterminism), the agent may see a different observation than recorded. A message-count check warns on divergence (`--strict-divergence` aborts instead). * **"Identical output" means a faithful continuation**, not a bit-identical result — the model samples, and no "original full run" exists past the timeout. The bar is: the stitched trajectory reads as one continuous run, as if the timeout had been larger. * Re-running the episode's commands costs wall-clock time (model latency is skipped, since recorded responses are served instantly). # Environment plane Source: https://docs.benchflow.ai/environment-plane # The Environment plane The **Environment plane** is the stateful world the agent acts in — Han's "S" in `E = {T, H, V, S, C}`. It is one of BenchFlow's four swappable planes (Sandbox, Agent, Environment, Reward). See [`architecture.md`](./architecture.md), "The Environment plane & the manifest". A benchmark author never subclasses the framework. They write one file — an **`environment.toml` manifest** — and the default adapter (`ManifestEnvironment`) runs it on any Sandbox provider. The manifest is the entire integration surface. ## The manifest schema The manifest's keys live under an `[environment]` table. ### `[environment]` | Field | Type | Default | Meaning | | ---------------- | ------------------------------ | ------------ | ----------------------------------------------------------------------------------------------------- | | `name` | str | — (required) | Environment / benchmark name. | | `image` | str | `None` | A ready-to-run image. Set this **or** `base_image`. | | `base_image` | str | `None` | Image that per-task images build `FROM` (smolclaws-style). | | `ports` | list\[int] | `[]` | Ports the environment exposes (in addition to service ports). | | `owns_lifecycle` | bool | `true` | `true` — the image entrypoint starts the services. `false` — the framework starts the `[[services]]`. | | `keep_alive` | bool | `true` | Keep the environment up for the whole rollout. | | `isolation` | `"per_task"` \| `"persistent"` | `"per_task"` | `per_task` — a fresh environment per episode. `persistent` — cross-episode state. | Exactly one of `image` / `base_image` must be set. When `owns_lifecycle` is `false` the manifest must declare `[[environment.services]]`; when it is `true` it must not. ### `[environment.task_selection]` | Field | Type | Default | Meaning | | ------------- | -------------------------- | --------------------- | -------------------------------------------------------------------------------------------------------------------- | | `mechanism` | `"image"` \| `"env_var"` | `"env_var"` | `image` — the task's seed data is baked into a per-task image. `env_var` — one image, the task id passed at runtime. | | `key` | str | `"BENCHFLOW_TASK_ID"` | Env var name (when `mechanism = "env_var"`). | | `inject_into` | `"entrypoint"` \| `"exec"` | `"entrypoint"` | `entrypoint` reaches PID 1; `exec` does not. | ### `[[environment.services]]` An array — one table per service the framework starts (only when `owns_lifecycle = false`). It is the declarative replacement for the hard-coded `SERVICES` dict in `benchflow/sandbox/services.py`. | Field | Type | Default | Meaning | | ------------- | ---- | ------------ | ------------------------------- | | `name` | str | — (required) | Service name. | | `command` | str | — (required) | Full start command. | | `port` | int | — (required) | Port the service listens on. | | `health_path` | str | `"/health"` | HTTP path probed for readiness. | ### `[environment.readiness]` | Field | Type | Default | Meaning | | ------------- | ---------- | ------- | ------------------------------------------------------------ | | `http` | list\[str] | `[]` | Explicit HTTP probes. When empty, derived from the services. | | `tcp` | list\[int] | `[]` | TCP-connect probes. | | `timeout_sec` | int | `120` | How long to wait for readiness before failing the rollout. | ### `[environment.forward_env]` | Field | Type | Default | Meaning | | ------ | ---------- | ------- | ------------------------------------------------------- | | `keys` | list\[str] | `[]` | Host env vars forwarded into the environment container. | ### `[environment.state]` Present only for an environment that supports **roll-back** — `snapshot` / `restore`. Absent this table, the environment is treated as stateless and `snapshot`/`restore` raise `RuntimeError`. | Field | Type | Default | Meaning | | ------- | ---------- | ---------- | ---------------------------------------------------------------------------- | | `kind` | `"sqlite"` | `"sqlite"` | State backend. Only SQLite is supported today. | | `paths` | list\[str] | `[]` | The database files to capture and restore (one snapshot covers all of them). | ## Worked example — ClawsBench `benchmarks/clawsbench/environment.toml` — the internal-dogfood stateful multi-service benchmark (mock Gmail / Slack / Calendar / Docs / Drive): ```toml theme={null} [environment] name = "clawsbench" base_image = "kywch/smolclaws-base:latest" owns_lifecycle = false isolation = "per_task" [environment.task_selection] mechanism = "image" [environment.readiness] timeout_sec = 60 [environment.forward_env] keys = ["ANTHROPIC_API_KEY"] [[environment.services]] name = "gmail" command = "claw-gmail --db /data/gmail.db serve --host 0.0.0.0 --port 9001 --no-mcp" port = 9001 # ... slack (9002), gcal (9003), gdoc (9004), gdrive (9005) ``` One manifest serves the whole benchmark even though smolclaws builds a per-task image carrying only a subset of the services: `ManifestEnvironment` probes each service's entry point with `--help` and starts only the services whose package is actually installed in this per-task image. ## Worked example — chi-bench `benchmarks/chi-bench/environment.toml` — the *other* topology, and the external proof that a heavy environment onboards untouched. chi-bench is a \~25k-LOC healthcare simulator that ships **one** ready-to-run image whose entrypoint starts its own services, so the manifest declares no `[[services]]`: ```toml theme={null} [environment] name = "chi-bench" image = "chi-bench:latest" owns_lifecycle = true isolation = "per_task" ports = [8020, 8023, 8100, 8200] [environment.task_selection] mechanism = "env_var" key = "CHI_BENCH_TASK_ID" inject_into = "entrypoint" [environment.readiness] http = ["http://localhost:8023/health"] timeout_sec = 120 [environment.forward_env] keys = ["ANTHROPIC_API_KEY"] ``` This \~25-line manifest is the *entire* framework-integration surface: chi-bench's image, Dockerfile, and entrypoint are unmodified, and the \~920 LOC of Harbor coupling it previously carried collapses into the manifest. ClawsBench (`base_image` + framework-started `[[services]]`) and chi-bench (`image` + `owns_lifecycle = true`) are the two topologies behind one contract. See [`benchmarks/chi-bench/README.md`](../benchmarks/chi-bench/README.md) for the field-by-field mapping. ## How it runs `ManifestEnvironment` runs the **in-sandbox topology** (the architecture's core): the services run inside the rollout's own sandbox, so the agent reaches them on `localhost`. During a rollout: 1. `Rollout.start()` provisions the environment — starts the declared services inside the sandbox. 2. It gates on `readiness()` — the agent never runs before the environment is healthy. 3. `Rollout.cleanup()` tears the environment down. Run one task or a task directory against an environment manifest with `bench eval create --tasks-dir ...`. `--environment-manifest` applies the Environment-plane manifest to every rollout in the Job pipeline. ```bash theme={null} # one task bench eval create --tasks-dir benchmarks/clawsbench/tasks/ \ --environment-manifest benchmarks/clawsbench/environment.toml \ --agent claude-agent-acp --model claude-haiku-4-5 # task directory bench eval create --tasks-dir benchmarks/clawsbench/tasks \ --environment-manifest benchmarks/clawsbench/environment.toml \ --agent claude-agent-acp --model claude-haiku-4-5 ``` YAML configs may declare the same seam with `environment_manifest: ` at the top level so the batch run is reproducible from disk. `--environment-manifest` is distinct from `--sandbox`: the sandbox is *where* it runs (the Sandbox plane); the environment manifest is *the world* (the Environment plane). ## Exporting for training A scored rollout's trajectory exports to the Verifiers / ORS dataset format that prime-rl ingests — `benchflow.trajectories.export`: ```python theme={null} from benchflow.trajectories.export import ( trajectory_to_verifiers_record, export_trajectories_to_jsonl, ) record = trajectory_to_verifiers_record( task_id="clawsbench/archive-alice", messages=trajectory_messages, verify_result=verify_result, model="claude-haiku-4-5", environment="clawsbench", ) export_trajectories_to_jsonl([record], "dataset.jsonl") ``` Each line is one record: `prompt`, `completion`, `reward`, `metrics`, `is_completed`, `is_truncated`, `example_id`, `info` — the shape pinned against the Verifiers `RolloutOutput` type. ## Roll-back — `snapshot` / `restore` `snapshot` / `restore` are **real**. For an environment that declares an `[environment.state]` table, `snapshot()` copies each declared SQLite file with `sqlite3 .backup` (a consistent online backup) into a per-snapshot directory inside the sandbox, and `restore(snap)` copies the captured files back over the live paths. This is the substrate `Rollout.branch()` runs on: a branch quiesces the agent and services, restores a snapshot, and explores an alternative continuation. An environment with no `[environment.state]` table is stateless — `snapshot`/`restore` raise `RuntimeError`. ## Reset — `reset` `reset` returns the environment to the per-task baseline so it can be reused for a fresh episode without tearing down the sandbox (distinct from `restore`, which rolls back to an arbitrary snapshot). For an environment that declares an `[environment.state]` table, `provision` captures a baseline; `reset` then stops the framework-started services, restores the baseline, and restarts the services. For an `owns_lifecycle = true` manifest the framework cannot restart entrypoint-owned services; `reset` is then a no-op (and the host must recycle the container for a hard reset). ## Not yet implemented `ManifestEnvironment` does not exercise: * **Sidecar / shared-fleet topology** — host-exposed ports, `AccountBroker`. # Getting started Source: https://docs.benchflow.ai/getting-started # Getting started A 5-minute path from install to first eval. ## Prerequisites * Python 3.12+ * [`uv`](https://docs.astral.sh/uv/) * Docker for local sandboxes, `pip install benchflow[sandbox-daytona]` + `DAYTONA_API_KEY` for Daytona cloud runs, or `pip install benchflow[sandbox-modal]` for Modal-backed runs * An API key or subscription/OAuth auth for at least one agent (see below) ## Install `0.6.0` is on PyPI. Install (or upgrade) with uv or pip: ```bash theme={null} uv tool install --prerelease allow benchflow # add --upgrade to refresh pip install --pre --upgrade benchflow # pip equivalent ``` The `--prerelease allow` (uv) / `--pre` (pip) flag is required for BenchFlow's pinned LiteLLM release-candidate dependency, not for benchflow itself (`0.6.0` is a final release). If `uv` reports `Executables already exist: bench, benchflow`, rerun with `--force` to replace older non-`uv` entrypoints. Confirm with `bench --version`. See [Release channels](./release.md) for the full command matrix. This gives you the `benchflow` (alias `bench`) CLI plus the Python SDK. To install for editable development: ```bash theme={null} git clone https://github.com/benchflow-ai/benchflow cd benchflow uv sync --extra dev --locked ``` ## Auth: OAuth, long-lived token, or API key You don't need an API key if you're a Claude / Codex / Gemini subscriber. Three options, pick one per agent: ### Option 1 — Subscription OAuth from host CLI login If you've logged into the agent's CLI on your host (`claude login`, `codex --login`, `gemini` interactive flow), benchflow picks up the credential file and copies it into the sandbox. No API key billing. | Agent | How to log in on the host | What benchflow detects | Replaces env var | | ------------------ | -------------------------------- | ----------------------------- | ------------------- | | `claude-agent-acp` | `claude login` (Claude Code CLI) | `~/.claude/.credentials.json` | `ANTHROPIC_API_KEY` | | `codex-acp` | `codex --login` (Codex CLI) | `~/.codex/auth.json` | `OPENAI_API_KEY` | | `gemini` | `gemini` (interactive login) | `~/.gemini/oauth_creds.json` | `GEMINI_API_KEY` | When benchflow finds the detect file, you'll see: ``` Using host subscription auth (no ANTHROPIC_API_KEY set) ``` ### Option 2 — Long-lived OAuth token (CI / headless) For CI pipelines, scripts, or anywhere the host can't run an interactive browser login, generate a 1-year OAuth token with `claude setup-token` and export it: ```bash theme={null} claude setup-token # walks you through browser auth, prints a token export CLAUDE_CODE_OAUTH_TOKEN= ``` benchflow auto-inherits `CLAUDE_CODE_OAUTH_TOKEN` from your shell into the sandbox; the Claude CLI inside reads it directly. Same auth precedence as plain `claude` ([Anthropic docs](https://code.claude.com/docs/en/authentication#authentication-precedence)): API keys override OAuth tokens, so unset `ANTHROPIC_API_KEY` if you want the token to win. `claude setup-token` only authenticates Claude. Codex can also use a provided subscription access token, such as `CODEX_ACCESS_TOKEN` from a host/orchestrator integration; benchflow passes it through to Codex without copying `~/.codex/auth.json`. Gemini does not have an equivalent today — use Option 1 (host login) or Option 3 (API key). ### Option 3 — API key Set the API-key env var directly. Works with every agent: ```bash theme={null} export ANTHROPIC_API_KEY=sk-ant-... export OPENAI_API_KEY=sk-... export CODEX_API_KEY=sk-... # Codex alias for OPENAI_API_KEY export GEMINI_API_KEY=... export LLM_API_KEY=... # OpenHands / LiteLLM-compatible providers export AZURE_API_KEY=... export AZURE_API_ENDPOINT=https://.openai.azure.com/ ``` benchflow auto-inherits well-known API key env vars from your shell into the sandbox. Provider-prefixed models can use credentials that differ from the agent's native default auth. For Azure Foundry, use models such as `azure-foundry-openai/gpt-5.5` or `azure-foundry-anthropic/claude-opus-4-5`; benchflow derives the Azure resource from `AZURE_API_ENDPOINT` and routes the selected agent through a generated LiteLLM gateway config. Several providers with user-supplied endpoints — `deepseek`, `glm`, `kimi`, `minimax`, `hunyuan`, and others — follow the `_API_KEY` + `_BASE_URL` convention; providers with fixed endpoints (such as `zai` or `openai`) need only the API key. For example, `deepseek/` reads: ```bash theme={null} export DEEPSEEK_API_KEY=... export DEEPSEEK_BASE_URL=https://api.deepseek.com ``` If the base URL is missing, the run fails with `Provider 'deepseek' for model 'deepseek/' requires DEEPSEEK_BASE_URL to build the provider base URL.` These variables must be **exported** to reach the benchflow runtime — a plain shell assignment or a `source .env` without `export` stays local to your shell and never reaches the `bench` process. The portable pattern for a `.env` file: ```bash theme={null} set -a; source .env; set +a bench eval create ... ``` (benchflow also picks up well-known credential keys from a `.env` file in the current directory; exporting works from any directory.) ### Precedence If multiple credentials are set, benchflow / the agent CLI uses provider-specific credentials selected by the model prefix first, then the agent's native auth precedence. For Claude, native auth is (high to low): cloud provider creds → `ANTHROPIC_AUTH_TOKEN` → `ANTHROPIC_API_KEY` → `apiKeyHelper` → `CLAUDE_CODE_OAUTH_TOKEN` → host subscription OAuth. To force a lower-priority option, unset the higher one in your shell before running. ## Run your first eval ```bash theme={null} # Single task from a local directory GEMINI_API_KEY=... bench eval create \ --tasks-dir tasks/edit-pdf \ --agent gemini \ --model gemini-3.1-pro-preview \ --sandbox docker # Single task with mounted skills GEMINI_API_KEY=... bench eval create \ --tasks-dir tasks/edit-pdf \ --agent gemini \ --model gemini-3.1-pro-preview \ --sandbox daytona \ --skill-mode with-skill \ --skills-dir tasks/edit-pdf/environment/skills \ --agent-env BENCHFLOW_SKILL_NUDGE=name # A whole batch from YAML config bench eval create --config benchmarks/harvey-lab/harvey-lab-gemini-flash-lite.yaml # Batch over a local tasks directory with concurrency GEMINI_API_KEY=... bench eval create \ --tasks-dir tasks \ --agent gemini --model gemini-3.1-pro-preview --sandbox daytona --concurrency 32 # List the registered agents bench agent list ``` `bench eval create` is the primary command for running evaluations — it works for single tasks, batch runs, and remote repos. Use `--tasks-dir ` for a local directory or `--config ` for a YAML config. You can also fetch tasks straight from a remote repo with `--source-repo --source-path `, but note that this clones the full repository (`git clone --depth 1` into `.cache/datasets///` under the enclosing git repo root, or the current directory when you run outside one) — large for big task repos. To download only the task you need, use a sparse checkout and point `--tasks-dir` at it: ```bash theme={null} git clone --depth 1 --filter=blob:none --sparse https://github.com/benchflow-ai/skillsbench cd skillsbench && git sparse-checkout set tasks/edit-pdf bench eval create --tasks-dir tasks/edit-pdf --agent gemini --model gemini-3.1-pro-preview ``` When you mount skills, use `BENCHFLOW_SKILL_NUDGE=name` as the default docs option. See [Architecture: skill loading](./architecture.md#skill-loading) for how mounted skills reach the agent and how `name`, `description`, and `full` differ. ### Where results land Each run writes under `--jobs-dir` (default `jobs/`): ``` / summary.json # copy of the latest job summary (overwritten by the next run) / # job directory, named by start time summary.json # job-level aggregate __/ # one rollout: task name + 8-char id result.json # rollout summary: rewards, errors, token usage/cost rewards.jsonl # reward record for this rollout timing.json # per-phase timing breakdown prompts.json # prompts sent to the agent trajectory/ acp_trajectory.jsonl # full agent trace (ACP events) llm_trajectory.jsonl # raw provider requests/responses (when the usage-tracking proxy captured exchanges) trainer/ verifiers.jsonl # trainer-ready scored trajectory (Verifiers/ORS record) atif.json # ATIF trajectory record (omitted if the trajectory is empty) adp.jsonl # ADP trajectory record verifier/ ctrf.json # CTRF test report (when test.sh emits one) reward.txt # raw verifier reward (0.0-1.0) test-stdout.txt # verifier stdout ``` ### Reading results Exit code 0 means the pipeline completed — it is not a pass/fail signal. A rollout whose reward is below the pass threshold still exits 0 and prints `[FAIL]` with `Score: 0/1`: `Score` is pass-threshold aggregation (a task counts as passed only at reward 1.0), while `reward` — in `result.json` and `verifier/reward.txt` — is the raw verifier value. Config errors (unknown agents, missing credentials) exit 1, and so do runs with agent or verifier errors. CLI usage errors (bad flags) exit 2. The Docker sandbox needs the Docker daemon running. There is no up-front check — if the daemon is down the run fails partway through rather than at startup, so start Docker before `bench eval create --sandbox docker`. ## Run from Python The CLI is a thin shim over the Python API. For programmatic use: ```python theme={null} import benchflow as bf from benchflow import RolloutConfig, Scene from benchflow._utils.benchmark_repos import resolve_source config = RolloutConfig( task_path=resolve_source("benchflow-ai/skillsbench", path="tasks/edit-pdf"), scenes=[Scene.single(agent="gemini", model="gemini-3.1-pro-preview")], environment="docker", ) result = await bf.run(config) print(result.rewards) # {'reward': 1.0} print(result.n_tool_calls) ``` `Rollout` is decomposable — invoke each lifecycle phase individually for custom flows. See [Concepts: rollout lifecycle](./concepts.md#rollout-lifecycle). ## What to read next | If you want to… | Read | | --------------------------------------------------------------------- | ----------------------------------------------------- | | Understand how BenchFlow runs *any* benchmark (the three-layer model) | [Run any benchmark](./running-any-benchmark.md) | | Understand the model — Rollout, Scene, Role, Verifier | [Concepts](./concepts.md) | | Author a task | [Task authoring](./task-authoring.md) | | Run multi-agent patterns (coder/reviewer, simulated user, BYOS) | [Use cases](./use-cases.md) | | Run multi-round single-agent (progressive disclosure) | [Progressive disclosure](./progressive-disclosure.md) | | Evaluate skills, not tasks | [Skill eval](./skill-eval.md) | | Understand the security model | [Sandbox hardening](./sandbox-hardening.md) | | CLI flags + commands | [CLI reference](./reference/cli.md) | | Python API surface | [Python API reference](./reference/python-api.md) | # Integration tests Source: https://docs.benchflow.ai/integration-tests # Integration Tests On-demand end-to-end tests that validate BenchFlow against real benchmark suites. Not part of CI — invoke manually before trial-ready releases and before large runtime refactors. The core matrix runs 9 SkillsBench tasks across all 9 registered agents on Daytona. Release readiness also requires smoke coverage for the current adapter release set, the current feature release set, hosted environment compatibility, and Terminal-Bench-style tasks so BenchFlow keeps running existing suites even as public API names move to Rollout/Sandbox terminology. ## Prerequisites Install Daytona sandbox support for local integration runs: ```bash theme={null} uv sync --extra dev --extra sandbox-daytona --locked ``` | Variable | Required for | | ------------------------------------------------ | --------------------------------------------- | | `GEMINI_API_KEY` (or `GOOGLE_API_KEY`) | gemini, pi-acp, openclaw, opencode, openhands | | `DAYTONA_API_KEY` | all agents (sandbox) | | `CLAUDE_CODE_OAUTH_TOKEN` or `ANTHROPIC_API_KEY` | claude-agent-acp | | `OPENAI_API_KEY` | codex-acp | | `XIAOMI_API_KEY` + `XIAOMI_BASE_URL` | mimo | This table covers the default integration suite models. Provider-specific integration lanes may require different credentials; Azure Foundry lanes use `AZURE_API_KEY` plus `AZURE_API_ENDPOINT`. ## Quick Start ```bash theme={null} # All 8 agents in parallel (each runs 9 tasks concurrently on Daytona) export GEMINI_API_KEY=... DAYTONA_API_KEY=... CLAUDE_CODE_OAUTH_TOKEN=... OPENAI_API_KEY=... tests/integration/run.sh # Specific agents only tests/integration/run.sh gemini pi-acp claude-agent-acp # Review results from a previous run (no API calls) tests/integration/run.sh --check-only # Large validation override BENCHFLOW_INTEGRATION_CONCURRENCY=100 tests/integration/run.sh gemini ``` ## What It Does 1. **Resolves tasks** — downloads the full SkillsBench task set, then creates a symlinked subset of 9 selected tasks. 2. **Launches agents in parallel** — each agent is started as a background process running `bench eval create` with concurrency=64 by default. Set `BENCHFLOW_INTEGRATION_CONCURRENCY=100` for the large post-migration validation run. 3. **Waits and reports** — as each agent finishes, prints its score line. After all complete, runs `check_results.py` to validate output schema and print the results table. ## Release Readiness Coverage Before cutting a trial-ready release, every current release blocker must pass. If any adapter, feature, hosted environment board entry, or first-party sandbox in the current release set fails validation, do not publish the release. | Release blocker | Purpose | Minimum acceptance | | ----------------------------- | --------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | SkillsBench agent matrix | Validates the full agent × task pipeline on Daytona | 9 selected tasks across all credentialed agents produce valid `result.json`, trajectory output, and summary schema | | Adapter release set | Validates merged and open benchmark adapters preserve source-suite semantics | Each benchmark listed in [Running Adapted Benchmarks](./running-benchmarks.md#available-benchmarks), plus each open benchmark-adapter PR targeted at the release, has at least one representative smoke or parity run with verifier execution and reward schema validation | | Terminal-Bench smoke | Validates Terminal-Bench-style task packaging and shell verifier behavior | At least one representative task runs through `bench eval create`, verifier executes from the task tests, and sandbox hardening does not break normal task execution | | Trace-to-task | Validates `bench tasks generate` can turn real traces into runnable benchmark tasks | Generate from at least one local or JSONL trace and one HuggingFace/opentraces source, then run the generated task through `bench eval create` and validate the verifier outcome | | Agent decoupling | Validates agents are pluggable runtime targets rather than core-coupled implementations | Core import and task validation work without optional agent packages; at least two non-oracle agents run the same representative task through the same Rollout path | | Sandbox decoupling | Validates sandbox backends are pluggable and optional dependencies remain isolated | Core `import benchflow` works without optional sandbox extras; Docker and Daytona smoke runs use the same task contract; missing optional sandbox deps produce clear install guidance | | Release-gated sandbox support | Validates the release-quality sandbox backends in the current release gate | Docker and Daytona can run a representative task via `bench eval create --sandbox docker` and `--sandbox daytona` without special user intervention beyond each backend's documented auth; Modal remains optional follow-up evidence, not a release blocker | These blockers test benchmark-suite portability, not backward compatibility with old BenchFlow names. Public docs and new configs should use Rollout/Sandbox terminology. For the current release-prep sweep, the adapter release set is keyed by benchmark UID. The UID is derived from the BenchFlow task source as `{source.repo}:{source.path}@{source.ref}` so display names and PR titles are never the primary identity. | Benchmark UID | Name | Status | | -------------------------------------------------------------------- | ---------------------- | -------------------------------------------------------------------------- | | `benchflow-ai/benchmarks:datasets/harvey-lab/tasks@main` | Harvey LAB | Merged adapter | | `benchflow-ai/benchmarks:datasets/programbench/tasks@main` | ProgramBench | Merged adapter | | `benchflow-ai/skillsbench:tasks@main` | SkillsBench | Merged adapter | | `benchflow-ai/benchmarks:datasets/hilbench/tasks@main` | HILBench | Open adapter PR [#279](https://github.com/benchflow-ai/benchflow/pull/279) | | `benchflow-ai/benchmarks:datasets/opaquetoolsbench/tasks@main` | OpaqueToolsBench | Open adapter PR [#280](https://github.com/benchflow-ai/benchflow/pull/280) | | `benchflow-ai/benchmarks:datasets/continuallearningbench/tasks@main` | ContinualLearningBench | Open adapter PR [#283](https://github.com/benchflow-ai/benchflow/pull/283) | HILBench uses the Hugging Face dataset `ScaleAI/hil-bench` for task metadata, but its SWE image tarballs live in the Hugging Face bucket `ScaleAI/hil-bench-swe-images`. Adapter code should treat dataset `repo_or_db_download_link` values such as `hf://buckets/ScaleAI/hil-bench-swe-images/images/.tar.zst` as bucket objects and fetch them through `https://huggingface.co/buckets/ScaleAI/hil-bench-swe-images/resolve/images/.tar.zst`, not through dataset `hf_hub_download`. Hosted environment registries such as PrimeIntellect, OpenReward/ORS, and Harbor registry should not use benchmark UIDs unless their envs have been converted into checked-in BenchFlow task sources. Track those with `env_uid` plus the canonical `hub_url` in the compatibility board, and record a separate `benchflow_uid` only after conversion. The current hub URLs are `https://openreward.ai/environments`, `https://hub.harborframework.com/`, and `https://app.primeintellect.ai/dashboard/environments?ex_sort=by_sections`. The current OpenReward compatibility-board selections are `openreward:GeneralReasoning/KellyBench@be14865a-3c70-422e-a2ba-f45c132cd29a` for long-horizon sandbox/tool use and `openreward:GeneralReasoning/CTF@fcfcd0ef-1298-40e9-9492-83628fd98a1c` for security sandbox coverage. The current PrimeIntellect selections are `primeintellect:primeintellect/reverse-text@0.1.4` for lightweight/no-secret inventory and `primeintellect:primeintellect/math-python@0.1.10` for tool-use plus Prime sandbox coverage. The current Harbor selections are `harbor:terminal-bench/adaptive-rejection-sampler@69671fbaac6d67a7ef0dfec016cc38a64ef7a77c` for Terminal-Bench-style packaging and `harbor:binary-audit/caddy-backdoor-detect@75f3e6e331776b80f77faa3d2ff80627b8b5d069` for security-style adaptation. OpenReward exposes stable environment IDs and `updated_at` timestamps in its catalog rather than semantic environment versions, so the compatibility board uses the environment ID in `env_uid` and records `updated_at` separately. OpenReward session-level smoke may still require account credits even when catalog inventory succeeds. For the current release-prep sweep, the feature release set includes: | Feature | Scope | | ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Trace-to-task | `bench tasks generate` from local sessions, JSONL traces, and HuggingFace/opentraces datasets | | Agent decoupling | Agents are selected and configured outside core benchmark/runtime logic | | Sandbox decoupling | Sandbox backends are optional, selectable, and isolated behind the Sandbox contract | | Release-gated sandboxes | Docker and Daytona are release-gated through `--sandbox docker` and `--sandbox daytona`; Modal may remain selectable but is not a current release blocker | | Future hard-isolation backlog | Firecracker and Kubernetes are tracked in the backlog profile, not the current release gate. Promote them only after `--sandbox firecracker`, `--sandbox k8s`, and the `--sandbox kubernetes` alias are implemented with install guidance and smoke evidence | ## Large Test Suite Structure The large suite should be built from explicit axes and named lanes, not a full Cartesian product. Agent × model × sandbox × task multiplies too quickly, so each lane must state what risk it covers. The declarative source of truth for the release-ready suite is [`tests/integration/suites/release.yaml`](../tests/integration/suites/release.yaml). Runners should load named lanes from that manifest rather than hardcoding matrix logic in shell scripts. Plan the release suite before running anything: ```bash theme={null} uv run python tests/integration/run_suite.py --list-lanes uv run python tests/integration/run_suite.py --list-profiles uv run python tests/integration/run_suite.py --profile near-term --dry-run uv run python tests/integration/run_suite.py --profile release-gated-cli --dry-run --fail-on-todo uv run python tests/integration/run_suite.py --profile hosted-envs --dry-run uv run python tests/integration/run_suite.py --profile full-release --dry-run --fail-on-todo # The backlog profile is expected non-zero until Firecracker/K8s are promoted. uv run python tests/integration/run_suite.py --profile backlog --dry-run --fail-on-todo uv run python tests/integration/run_suite.py --lane shared-sandbox-smoke --dry-run ``` `run_suite.py` starts dry-run-first and wires execution lane by lane. Adapter, hosted-env, and trace-to-task evidence are currently executable: ```bash theme={null} # Regenerate ENG-93 trace-to-task evidence, including Docker oracle evals. uv run python tests/integration/run_suite.py --lane trace-to-task-e2e --execute-trace-evidence --run-trace-eval # Validate adapter evidence from checked-in parity artifacts, a SkillsBench result, # and one worktree per open adapter PR. uv run python tests/integration/run_suite.py --lane adapter-release-set --execute-adapter-evidence \ --skillsbench-result dogfood/.../result.json \ --open-pr-root HILBench=/path/to/pr-279-worktree \ --open-pr-root OpaqueToolsBench=/path/to/pr-280-worktree \ --open-pr-root ContinualLearningBench=/path/to/pr-283-worktree # Validate hosted env hub metadata and regenerate Harbor inventory evidence. uv run python tests/integration/run_suite.py --lane hosted-env-compatibility-board --execute-hosted-env-evidence ``` Trace-to-task evidence writes generated tasks, oracle job outputs, and `trace-evidence.json` under `dogfood/2026-05-19-trace-to-task-e2e/` by default. The evidence directory is declared in [`tests/integration/suites/release.yaml`](../tests/integration/suites/release.yaml) and can be overridden with `--trace-evidence-dir`. Hosted-env evidence writes `hosted-env-evidence.json` plus Harbor registry JSONL inventory under `dogfood/2026-05-19-release-gate/hosted-envs/` by default. OpenReward and PrimeIntellect remain hub-metadata checks until account/credited hosted eval support is available; Harbor has a public registry inventory path. SkillsBench-vs-Harbor parity is an offline checker over existing artifacts. It does not clone `benchflow-ai/skillsbench-trajectories` during the release run because that baseline repository is large. Supply a local checkout or extracted artifact root pinned to `2d86fe82f6a06f7c7b3a22a3ae90d554d0e9655c`: ```bash theme={null} uv run python tests/integration/run_suite.py --lane skillsbench-harbor-parity \ --execute-skillsbench-harbor-parity \ --skillsbench-harbor-benchflow-root jobs/integration-/ \ --skillsbench-harbor-baseline-root /path/to/skillsbench-trajectories/shenghan/gemini-cli/gemini3flash/noskills \ --skillsbench-harbor-task jax-computing-basics \ --skillsbench-harbor-task python-scala-translation \ --skillsbench-harbor-task jpg-ocr-stat \ --skillsbench-harbor-task grid-dispatch-operator \ --skillsbench-harbor-task threejs-to-obj \ --skillsbench-harbor-task data-to-d3 \ --skillsbench-harbor-task lake-warming-attribution \ --skillsbench-harbor-task weighted-gdp-calc \ --skillsbench-harbor-task shock-analysis-supply ``` The checker normalizes the expected schema differences explicitly: Harbor rewards come from `verifier_result.rewards.reward` and ATIF trajectories from `agent/trajectory.json`, while BenchFlow rewards come from `rewards.reward` and ACP trajectories from `trajectory/acp_trajectory.jsonl`. It fails on missing tasks, malformed artifacts, missing trajectories, unseen task outcomes, per-task reward movement outside the Harbor observed range, and aggregate outcome/reward-rate drift over the configured thresholds. To refresh the Harbor pin, fetch the baseline repository, inspect the candidate run root, run the parity checker against known BenchFlow evidence, and update `tests/integration/suites/release.yaml` plus the command above only after the new baseline is accepted: ```bash theme={null} git -C /path/to/skillsbench-trajectories fetch origin main git -C /path/to/skillsbench-trajectories rev-parse origin/main ``` Use `--fail-on-todo` whenever a dry-run plan is being used as release evidence. Despite the name, this gate fails on unresolved TODOs and explicit `blocked_by` entries. The `near-term`, `release-gated-cli`, `hosted-envs`, and `full-release` profiles are expected to pass the gate today. The `backlog` profile is expected to fail this gate until the future Firecracker/Kubernetes security DinD lane is resolved; that failure tracks planned coverage, not a current release blocker. ### Run Tracking and Profiles Future release-suite runs should be tracked in Linear. Until that is wired into automation, `tests/integration/suites/release.yaml` is the source of truth for run intent and the job output paths are the evidence artifact. The `near-term` profile keeps release prep moving with a smaller plan: SkillsBench as the benchmark-suite focus, Daytona as the preferred cloud sandbox, and the adapter release set as the additional feature surface. This profile is not the full release gate; `full-release` includes every current release blocker. The `release-gated-cli` profile is the TODO-clean release-planning profile for the current release-gated CLI surface. It adds the shared sandbox smoke, Terminal-Bench-style shell verifier smoke, and decoupling checks over the Docker/Daytona surface without treating optional Modal or future Firecracker/Kubernetes support as release blockers. The `backlog` profile tracks planned validation lanes with concrete tasks and acceptance criteria. It currently holds the Firecracker/Kubernetes Docker-in-Docker smoke because those sandboxes are not part of the current release gate. Adapter coverage should stay intentionally small in the near-term profile: one representative smoke or parity task per adapter unless the result is ambiguous or fails in a way that needs narrowing. ### Axes | Axis | Examples | Notes | | ----------------- | ------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Agent | `gemini`, `claude-agent-acp`, `codex-acp`, `opencode`, `openhands` | Agents are selected independently from core runtime logic | | Model | `gemini-3.1-flash-lite-preview`, `gpt-5.4-nano`, Claude model IDs | Model coverage should be representative, not every model on every lane | | Sandbox | `docker`, `daytona`; optional `modal`; future `firecracker`, `k8s` / `kubernetes` | Docker/Daytona get the current release smoke lane; Modal is optional follow-up evidence; Firecracker/K8s stay in backlog hard-isolation lanes until implemented | | Task set | SkillsBench subset, Terminal-Bench smoke, security DinD smoke, generated trace tasks | Task sets should be named and versioned | | Benchmark adapter | Harvey LAB, ProgramBench, SkillsBench, open adapter PRs | Adapter lanes validate source-suite semantics and verifier behavior; benchmark identity is the source-derived UID, not the display name | | Hosted env hub | OpenReward environments, Harbor Hub, PrimeIntellect Environments Hub | Hosted env lanes track `env_uid` patterns, selected envs, and canonical hub URLs | | Feature | Trace-to-task, agent decoupling, sandbox decoupling | Feature lanes validate product/runtime behavior directly | ### Required Lanes | Lane | Axis slice | Purpose | | ------------------------------ | ------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------ | | Shared sandbox smoke | 1 boring representative task × oracle or one cheap agent × Docker/Daytona | Proves the same task contract works across every current release-gated sandbox | | SkillsBench agent matrix | 9 SkillsBench tasks × credentialed agents × default model × Daytona | Proves real agent execution across the registered-agent surface | | Adapter release set | 1 representative smoke/parity run per merged adapter and open adapter PR | Proves every adapter in the release set preserves source-suite semantics | | Hosted env compatibility board | OpenReward, Harbor Hub, PrimeIntellect hub entries × selected env UIDs or remaining selection TODOs | Proves hosted env catalogs are tracked as envs, not benchmark sources | | Terminal-Bench smoke | 1 Terminal-Bench-style task × Docker and one cloud sandbox | Proves shell verifier and packaging compatibility | | Trace-to-task e2e | local/JSONL trace + HuggingFace/opentraces source × generated tasks × Docker or Daytona | Proves generated tasks are runnable, not just emitted | | Decoupling checks | core import/task validation without optional packages; representative task on at least two agents and two sandboxes | Proves agents and sandboxes are not hard-coupled to core | ### Backlog Lanes | Lane | Axis slice | Promote when | | ------------------- | ----------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Security DinD smoke | `harbor:termigen-environments/docker_escape_privileged_container_medium@dc329464161db64b0c670f46fa39b62e4719dddd` × Firecracker/K8s | `bench eval create --sandbox firecracker` and `bench eval create --sandbox k8s` run the task without questions, Docker-in-Docker-style service/container operations work inside both sandboxes, and the verifier executes after sandbox hardening | ### Coverage Policy Release blockers are mandatory lanes inside the large suite. Backlog lanes must also name a real task and activation criteria, but they do not block the current release gate until promoted. Broader nightly or pre-merge runs can add pairwise coverage across agent/model/sandbox/task axes, but release decisions should stay tied to named lanes with explicit pass/fail evidence. ## Architecture ``` tests/integration/ ├── run.sh # Shell driver — parallel agent launch + wait ├── run_suite.py # Manifest-aware planner and lane evidence dispatcher ├── check_results.py # Result validator — schema checks + score table ├── check_adapter_evidence.py ├── check_trace_to_task_evidence.py ├── suites/ │ └── release.yaml # Declarative release-blocker suite └── configs/ # Per-agent YAML configs for standalone use ├── gemini.yaml ├── claude-agent-acp.yaml └── ... ``` Output lands in `jobs/integration//`: ``` jobs/integration/ ├── gemini/ │ ├── 2026-05-15__16-43-54/ # run directory │ │ ├── jax-computing-basics__abc123/ │ │ │ ├── result.json │ │ │ ├── trajectory/acp_trajectory.jsonl │ │ │ └── ... │ │ └── ... │ └── summary.json ├── claude-agent-acp/ │ └── ... └── .logs/ # per-agent stdout/stderr logs ├── gemini.log └── ... ``` ## Selected Tasks The 9 tasks (3 low / 3 medium / 3 high complexity): | Task | Complexity | | ------------------------ | ---------- | | jax-computing-basics | Low | | python-scala-translation | Low | | jpg-ocr-stat | Low | | grid-dispatch-operator | Medium | | threejs-to-obj | Medium | | data-to-d3 | Medium | | lake-warming-attribution | High | | weighted-gdp-calc | High | | shock-analysis-supply | High | ## Agents All 9 registered agents run by default: | Agent | Default Model | Notes | | ------------------ | ----------------------------- | ------------------------------------------------------ | | claude-agent-acp | claude-haiku-4-5-20251001 | Needs `CLAUDE_CODE_OAUTH_TOKEN` or `ANTHROPIC_API_KEY` | | codex-acp | gpt-5.4-nano | Needs `OPENAI_API_KEY` | | pi-acp | gemini-3.1-flash-lite-preview | | | openclaw | gemini-3.1-flash-lite-preview | | | gemini | gemini-3.1-flash-lite-preview | | | opencode | gemini-3.1-flash-lite-preview | | | harvey-lab-harness | gemini-3.1-flash-lite-preview | | | openhands | gemini-3.1-flash-lite-preview | | Agents missing credentials are automatically skipped. The notes above describe the default integration configs. Provider-prefixed model lanes can override the native agent auth requirements; Azure Foundry models use `AZURE_API_KEY` plus `AZURE_API_ENDPOINT`. ## Standalone YAML Configs Each agent has a YAML config in `configs/` with an `include` list restricting to the 9 selected tasks. These can be used directly with `bench eval create --config`: ```bash theme={null} uv run bench eval create --config tests/integration/configs/gemini.yaml ``` `run.sh` uses CLI arguments instead of these configs for parallel execution, but both approaches run the same 9 tasks at the active-dev concurrency of 64. The shell runner can be temporarily raised with `BENCHFLOW_INTEGRATION_CONCURRENCY=100` for the larger validation set. ## Result Validation `check_results.py` checks: * Every `result.json` has required fields (`task_name`, `agent`, `rewards`, `error`, `verifier_error`) * No infrastructure errors (sandbox failures vs. task failures) * `summary.json` exists with required keys (`total`, `passed`, `failed`, `errored`, `verifier_errored`, `score`) ```bash theme={null} # Run validator standalone uv run python tests/integration/check_results.py jobs/integration gemini pi-acp ``` ## Agent-as-Judge Verification `agent_judge.py` adds a second, model-based signal on top of the mechanical schema checks. Given a completed rollout directory, it reuses BenchFlow's own `call_judge` primitive (default model `gemini-3.1-flash-lite`) to grade whether the run is a trustworthy measurement: the agent genuinely attempted the task, the trajectory is coherent, and there is no obvious reward-hacking. The judge reads `result.json` plus the recorded `trajectory/acp_trajectory.jsonl`, and treats the trajectory as untrusted evidence rather than as instructions. The gate combines two requirements: 1. **Realness** — the run is REAL only when `n_tool_calls > 0`, token usage `> 0`, and the reward is non-null. These mechanical invariants hold independently of the judge: a judge pass cannot rescue an unreal run. 2. **Agent judge** — the judge must return a `pass` verdict. It is fail-closed: a missing provider SDK, an API error, or an unparseable or fieldless verdict all read as FAIL, never a silent pass. ```bash theme={null} # Judge a rollout dir (or a jobs root to search for the latest rollout) uv run python tests/integration/agent_judge.py jobs/integration-eval --json ``` The lightweight `.github/workflows/integration-eval.yml` workflow runs one small task through `bench eval create --agent openhands --model deepseek/deepseek-v4-flash --sandbox docker`, then runs the agent judge over the rollout and fails the job if the run is not REAL or the judge fails. It triggers on `workflow_dispatch` and nightly, and is capped at one task to keep the check cheap. It references the `DEEPSEEK_API_KEY`, `DEEPSEEK_BASE_URL`, and `GEMINI_API_KEY` repository secrets; the judge SDKs come from the `judge` extra (`uv sync --extra judge`). ## Robust integration suite (`tests/test_integration_suite.py`) The agent-judge gate above is the seed; `tests/test_integration_suite.py` grows it into a scenario suite whose checks are each grounded in a v0.6 dogfooding finding. Reusable building blocks live in `tests/integration/scenarios.py` (`run_eval`, `reward_of`, `synth_rollout`, the ATIF/ADP/secret-leak validators, `reaper_dryrun_issues`). The suite has two tiers. ### Deterministic integrity gates (run in the normal suite, no credentials) These exercise the gate over **synthetic rollout fixtures**, so they are fast, deterministic, and run in regular CI — closing the gaps a single happy-path live check leaves open: | Gate | What it pins | Dogfooding origin | | ------------------------------ | ---------------------------------------------------------------------------------------------------------------- | --------------------------------------------------- | | Realness gate | rejects a run that was scored but did no work (`n_tool_calls=0`), had no telemetry, was never scored, or errored | a "passed" run that is really resumed/empty results | | Fail-closed judge | a missing SDK, API error, or unparseable verdict reads FAIL, never a silent pass | judge infra error silently recorded as reward 0.0 | | Reward-hacking caught | a mechanically-REAL rollout (tool calls, tokens, reward 1.0) still fails when the judge flags verifier tampering | reward-hacking only the judge can see | | Example `judge.py` fail-closed | the shipped generated-skill-eval judge exits non-zero (writes no reward) when no LLM judge can run | ENG-254 reward-integrity fix | | Artifact integrity | ATIF is `ATIF-v1.x` with recognized step sources; ADP lines parse; no provider key leaks into any artifact | ATIF/ADP schema + secret-redaction findings | ```bash theme={null} # Runs with the normal test job; no keys, no sandbox. uv run pytest tests/test_integration_suite.py -m "not integration" ``` ### Live scenarios (`@pytest.mark.integration`, nightly / on demand) Real sandbox + provider runs, each skipping cleanly when its prerequisites (Docker daemon, `DAYTONA_API_KEY`, DeepSeek / Gemini keys) are absent: | Scenario | Assertion | Dogfooding origin | | ---------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------- | | `oracle_determinism_docker` | every oracle `solve.sh` self-scores reward 1.0 | the broken `3d-scan-calc` example oracle (ENG-256) | | `sandbox_parity_docker_daytona` | the same oracle task scores identically on Docker and Daytona | the rollout/daytona package split | | `agent_rollout_is_real_and_judged` | a real openhands + deepseek-v4-flash run is REAL and the agent judge runs over it; the verdict is recorded, not gated, since model success is stochastic | resumed/empty/idle-timeout shells | | `reaper_dryrun_is_safe` | `environment cleanup --dry-run` deletes nothing; foreign sandboxes are never reaped | destructive-reaper scoping | ```bash theme={null} export DEEPSEEK_API_KEY=... DEEPSEEK_BASE_URL=https://api.deepseek.com \ GEMINI_API_KEY=... DAYTONA_API_KEY=... uv run pytest tests/test_integration_suite.py -m integration ``` ## DeepSeek + `deepagents` harness and judge hardening The agent-as-judge is only as good as the adversarial behavior it has been stress-tested against. To harden it on a *different* harness than BenchFlow's ACP agents, `tests/integration/deepagents_harness.py` runs a **DeepSeek deep agent** in the [`deepagents`](https://github.com/langchain-ai/deepagents) (LangChain) framework: `create_deep_agent` drives `deepseek-v4-flash` with real shell/file tools that execute **inside a network-isolated Docker container** (the agent loop runs on the host; only its tools touch the sandbox), and the run is recorded as a BenchFlow rollout the same realness gate + LLM judge grade. Install with `uv sync --extra deepagents`. `run_deepagent(..., extra_system=...)` is steerable, so the same harness produces both genuine solves and reward-hacking attempts — the adversarial trajectories synthetic fixtures can't. ### What the dogfooding rounds found (and fixed) Iterating the judge against genuine + adversarial deepseek/deepagents rollouts surfaced two robustness gaps, both now fixed in `agent_judge.py`: 1. **Prompt-injection via tool observations.** A genuine run where the agent `cat`'d a file containing `<>` was wrongly failed — the judge treated untrusted *observation* text as an instruction. The prompt now separates the agent's **own actions** (messages + tool calls) from **observations** (untrusted environment data the agent merely read) and judges only the former, ignoring injected text in observations while still flagging an agent that *emits* a manipulation attempt. 2. **A tamper buried past the truncation window.** The judge only sees a head+tail excerpt; a `> tests/run_all.sh` tamper in the middle of a long genuine-looking run was missed. `_scan_verifier_tamper` now scans the **full** trajectory for write/delete/chmod of score-defining files (verifier, test, grader, reward files) and surfaces them to the judge via `flagged_verifier_actions`, so a truncated excerpt can't hide a tamper. Read-only inspection (running the tests) is not flagged, and the judge still allows a *legitimate* test edit the task asked for. `tests/test_judge_robustness.py` codifies this: deterministic scanner tests (normal CI) plus a live battery (`@pytest.mark.integration`) of honest runs (incl. a genuine failure and an injection-in-observation) and hacks (verifier tamper, test deletion, hardcoding, reward-file write, buried tamper) that the judge must classify correctly. The deepseek/deepagents end-to-end scenario lives in `test_integration_suite.py::test_deepagents_deepseek_rollout_is_real_and_judged`. # Llm judge Source: https://docs.benchflow.ai/llm-judge # LLM-as-Judge Verifier Use an LLM to evaluate agent outputs against a rubric instead of deterministic tests. *** ## When to use LLM-as-judge Use LLM-as-judge when the task output is subjective, open-ended, or hard to verify with unit tests — legal analysis, code review quality, document drafting, research summaries. For tasks with a clear right answer (e.g. "write fizzbuzz"), stick with deterministic `test.sh` verifiers. BenchFlow's LLM judge supports: * **First-class `[verifier]` type** — `type = "llm-judge"` in `task.toml`, no `test.sh` needed * **Multi-criterion rubrics** with binary, likert, and numeric scoring * **Per-criterion weights** for non-uniform importance * **Dense reward events** emitted per criterion during evaluation * **Multi-provider routing** across Anthropic, OpenAI, and Google models * **Configurable aggregation** (weighted mean, all-pass, any-pass, threshold) The judge is a **first-class verification method** alongside the deterministic `test.sh` verifier. A task selects it with one line of config — the framework handles deliverable collection, prompting, provider routing, retries, and reward aggregation. *** ## Quick start ### 0. Install the judge provider SDKs The judge calls the Anthropic, OpenAI, and Google SDKs — these are **not** installed by default. Install the `judge` extra (you only need at least one provider's SDK for the model you use, but the extra ships all three): ```bash theme={null} # in a checkout uv sync --extra judge # or as an installed tool uv tool install --prerelease allow 'benchflow[judge]==0.6.0' # or with pip pip install 'benchflow[judge]' ``` If no provider SDK is installed, the judge cannot run: the verifier raises a **verifier error** (the rollout is marked errored) rather than silently recording a reward of `0.0` — a missing dependency is an environment failure, not a score. ### 1. Select the judge verifier in `task.toml` ```toml theme={null} [verifier] type = "llm-judge" timeout_sec = 600 [verifier.judge] model = "claude-sonnet-4-6" # judge model (provider routed from prefix) rubric_path = "tests/rubric.toml" # rubric file, relative to the task dir input_dir = "/app" # sandbox dir holding agent deliverables # API keys for the judge — resolved from the host environment / .env [verifier.env] ANTHROPIC_API_KEY = "${ANTHROPIC_API_KEY}" ``` That's the entire verifier. There is **no `tests/test.sh`** to write — the `Verifier` downloads the agent's deliverables from `input_dir`, scores them against the rubric, and writes `reward.json` itself. ### 2. Write a `rubric.toml` Place it where `rubric_path` points (by convention `tests/rubric.toml`): ```toml theme={null} [[criterion]] name = "accuracy" description = "The response accurately addresses the question with correct facts" type = "binary" weight = 3.0 [[criterion]] name = "clarity" description = "The response is well-organized and easy to understand" type = "likert" points = 5 weight = 1.0 [scoring] aggregation = "weighted_mean" ``` A Harvey LAB style `rubric.json` works too — set `rubric_path = "tests/rubric.json"`: ```json theme={null} { "title": "Task Title", "criteria": [ {"id": "criterion-1", "title": "...", "match_criteria": "What constitutes a pass"} ] } ``` That's it. Run the task as usual — the reward is the proportion of criteria passed (or the configured aggregation), a partial float in `[0, 1]`. *** ## `[verifier]` reference | Field | Type | Default | Description | | ------------- | ------ | --------------- | ------------------------------------------------------ | | `type` | string | `"test-script"` | `"test-script"` (run `tests/test.sh`) or `"llm-judge"` | | `timeout_sec` | float | `600` | Overall verifier timeout | | `env` | table | `{}` | Env vars for the verifier — judge API keys go here | ### `[verifier.judge]` (used when `type = "llm-judge"`) | Field | Type | Default | Description | | ------------- | ------ | --------------------- | --------------------------------------------------------------------------------------- | | `model` | string | `"claude-sonnet-4-6"` | Judge model; provider routed from prefix | | `rubric_path` | string | `"tests/rubric.toml"` | Rubric file relative to the task dir (`.toml` or `.json`) | | `input_dir` | string | `"/app"` | Sandbox dir whose contents are graded | | `input_type` | string | `"deliverables"` | Only `"deliverables"` is supported — trajectory judging is not available at verify time | | `context` | string | `""` | Extra judge context (defaults to the task instruction) | *** ## Library use — `LLMJudgeRewardFunc` The judge is also a composable `RewardFunc`, usable directly or from a custom `test.sh` verifier: ```python theme={null} import asyncio from pathlib import Path from benchflow.rewards import LLMJudgeRewardFunc func = LLMJudgeRewardFunc(rubric_path=Path("rubric.toml")) score = asyncio.run(func.score(Path("/app"))) print(f"Score: {score:.2f}") ``` Auto-discovery — if `rubric.toml`/`rubric.json` is in the rollout directory or its parent, it's found automatically: ```python theme={null} func = LLMJudgeRewardFunc() score = asyncio.run(func.score(Path("/app"))) ``` *** ## rubric.toml reference ### `[judge]` section | Field | Type | Default | Description | | --------- | --------- | --------------------- | -------------------------------------------------------------------------------------------- | | `model` | string | `"claude-sonnet-4-6"` | LLM model for judging. Prefix with `anthropic/`, `openai/`, or `google/` to force a provider | | `mode` | string | `"individual"` | `"individual"` scores each criterion separately; `"batched"` is reserved for future use | | `files` | string\[] | `[]` | Default files to evaluate (fallback when a criterion doesn't specify its own) | | `timeout` | int | `120` | Timeout in seconds per judge call | ### `[[criterion]]` entries | Field | Type | Default | Description | | ------------- | --------- | ------------ | ------------------------------------------------------------------ | | `name` | string | — | Criterion identifier (falls back to first 40 chars of description) | | `description` | string | **required** | What the judge should evaluate | | `type` | string | `"binary"` | `"binary"`, `"likert"`, or `"numeric"` | | `weight` | float | `1.0` | Relative importance in aggregation | | `points` | int | `5` | Scale for likert type (1 to N) | | `min` | float | `0.0` | Minimum for numeric type | | `max` | float | `100.0` | Maximum for numeric type | | `files` | string\[] | `[]` | Specific files this criterion should evaluate | ### `[scoring]` section | Field | Type | Default | Description | | ------------- | ------ | ----------------- | --------------------------------------------------------- | | `aggregation` | string | `"weighted_mean"` | How to combine criterion scores | | `threshold` | float | `0.7` | Pass threshold (only used with `"threshold"` aggregation) | ### Score normalization Each criterion type normalizes its raw score to `[0, 1]`: | Type | Raw | Normalized | | --------- | ------------- | ------------------------------------------------ | | `binary` | pass/fail | `1.0` or `0.0` | | `likert` | 1–N integer | `(raw - 1) / (points - 1)` | | `numeric` | min–max float | `(raw - min) / (max - min)`, clamped to `[0, 1]` | ### Aggregation strategies | Strategy | Behavior | | --------------- | ------------------------------------------------------- | | `weighted_mean` | `sum(score × weight) / sum(weight)` — continuous reward | | `all_pass` | `1.0` if every criterion scores ≥ 0.5, else `0.0` | | `any_pass` | `1.0` if any criterion scores ≥ 0.5, else `0.0` | | `threshold` | `1.0` if weighted mean ≥ threshold, else `0.0` | *** ## Criterion types ### Binary (pass/fail) The judge decides whether the criterion is satisfied. The LLM returns `{"verdict": "pass", "reasoning": "..."}`. ```toml theme={null} [[criterion]] name = "has-executive-summary" description = "The document includes an executive summary in the first section" type = "binary" ``` ### Likert (scaled) The judge rates on a 1-to-N scale. The LLM returns `{"score": 4, "reasoning": "..."}`. ```toml theme={null} [[criterion]] name = "writing-quality" description = "Overall quality of prose — grammar, flow, and precision" type = "likert" points = 5 ``` A score of 3 on a 5-point scale normalizes to `(3-1)/(5-1) = 0.5`. ### Numeric (range) The judge assigns a value within a continuous range. The LLM returns `{"score": 75.0, "reasoning": "..."}`. ```toml theme={null} [[criterion]] name = "coverage-pct" description = "Percentage of key topics from the source material covered in the summary" type = "numeric" min = 0.0 max = 100.0 ``` *** ## Inline criteria (no TOML file) For programmatic use or Harvey LAB-style criteria, pass criteria directly: ```python theme={null} func = LLMJudgeRewardFunc( criteria=[ { "description": "The response is factually accurate", "type": "binary", "weight": 2.0, }, { "description": "The response addresses all parts of the question", "type": "binary", "weight": 1.0, }, ], judge_model="claude-sonnet-4-6", ) ``` Harvey LAB `match_criteria` keys are also supported: ```python theme={null} func = LLMJudgeRewardFunc( criteria=[ {"match_criteria": "Identifies the key risk factors", "type": "binary"}, {"match_criteria": "Provides supporting evidence", "type": "binary"}, ], ) ``` *** ## Dense reward events Each criterion emits a `RewardEvent` during evaluation, enabling per-criterion observability and training signal: ```python theme={null} func = LLMJudgeRewardFunc(rubric_path=Path("rubric.toml")) score = await func.score(rollout_dir) for event in func.events: print(f" {event.source}: {event.reward:.2f} (step {event.step})") ``` Output: ``` criterion:accuracy: 1.00 (step 0) criterion:clarity: 0.50 (step 1) criterion:completeness: 0.75 (step 2) ``` Events have type `"dense"`, a `reward` in `[0, 1]`, a `source` of `"criterion:{name}"`, and a `step` index. Events are cleared between `score()` calls. *** ## Multi-provider routing The judge model string determines which provider SDK is used: | Prefix | Provider | Auth env var | | ---------------------------------------- | --------- | ------------------------------------ | | `claude-*`, `anthropic/*` | Anthropic | `ANTHROPIC_API_KEY` | | `gpt-*`, `o1*`, `o3*`, `o4*`, `openai/*` | OpenAI | `OPENAI_API_KEY` | | `gemini*`, `google/*` | Google | `GOOGLE_API_KEY` or `GEMINI_API_KEY` | If the primary provider fails, the judge falls back through the other providers with retries and exponential backoff. The provider SDKs ship in the `judge` extra (`uv sync --extra judge`). If *none* are installed, the judge raises a verifier error instead of recording a reward — see [step 0](#0-install-the-judge-provider-sdks). *** ## Evaluation output After scoring, an `evaluation_details.json` is written to the rollout directory: ```json theme={null} { "score": 0.75, "n_passed": 2, "n_total": 3, "results": [ { "id": "accuracy", "description": "The response accurately addresses the question", "score": 1.0, "weight": 3.0, "verdict": {"verdict": "pass", "reasoning": "..."} }, { "id": "clarity", "description": "The response is well-organized", "score": 0.5, "weight": 1.0, "verdict": {"score": 3, "reasoning": "..."} } ] } ``` The `score` field is the actual aggregated score from the configured strategy, not `n_passed / n_total`. *** ## File discovery The judge automatically discovers deliverable files in the rollout directory. Supported formats: | Extension | Reader | Dependency | | ------------------------------ | --------------------- | ------------------------------------------------- | | `.txt`, `.md`, `.json`, `.csv` | Built-in | None | | `.docx` | pandoc or python-docx | `pandoc` (preferred) or `pip install python-docx` | | `.xlsx` | openpyxl | `pip install openpyxl` | | `.pptx` | markitdown | `pip install markitdown` | | `.pdf` | pdfplumber | `pip install pdfplumber` | Files larger than 50 MB are skipped. Hidden files (starting with `.`) and internal metadata files (`rubric.json`) are excluded. File content is truncated at 15,000 characters per file when sent to the judge. To scope a criterion to specific files: ```toml theme={null} [[criterion]] name = "memo-quality" description = "The legal memo follows IRAC structure" files = ["memo.docx", "analysis.md"] ``` *** ## Python API All rubric config types are importable from the top level: ```python theme={null} from benchflow import ( Criterion, JudgeConfig, LLMJudgeRewardFunc, RubricConfig, ScoringConfig, load_rubric, # dispatches on extension (.toml / .json) load_rubric_json, load_rubric_toml, ) # Load and inspect a rubric (TOML or Harvey LAB style JSON) rubric = load_rubric(Path("rubric.json")) print(f"Model: {rubric.judge.model}") print(f"Criteria: {len(rubric.criteria)}") for c in rubric.criteria: print(f" {c.id} ({c.type}, weight={c.weight})") ``` *** ## Worked example — Harvey LAB legal task A legal document analysis task scored entirely by config — no `test.sh`: ```toml theme={null} # task.toml [verifier] type = "llm-judge" timeout_sec = 600 [verifier.judge] model = "claude-sonnet-4-6" rubric_path = "tests/rubric.toml" input_dir = "/app" [verifier.env] ANTHROPIC_API_KEY = "${ANTHROPIC_API_KEY}" ``` ```toml theme={null} # tests/rubric.toml [judge] files = ["analysis.md"] [[criterion]] name = "key-terms-identified" description = "All material terms from the contract are identified and listed" type = "binary" weight = 2.0 [[criterion]] name = "risk-assessment" description = "Each identified risk includes severity rating and mitigation suggestion" type = "likert" points = 5 weight = 3.0 [[criterion]] name = "completeness" description = "Percentage of contract sections addressed in the analysis" type = "numeric" min = 0 max = 100 weight = 1.0 [scoring] aggregation = "weighted_mean" ``` The framework downloads the agent's deliverables from `/app`, grades each criterion, aggregates, and writes `reward.json` — no scripting required. *** ## Where to go next * [Concepts](./concepts.md) — the five primitives including Verifier * [Task authoring](./task-authoring.md) — `task.toml`, `tests/`, verifier contract * [Running benchmarks](./running-benchmarks.md) — Harvey LAB uses LLM-as-judge * [Python API reference](./reference/python-api.md) — `LLMJudgeRewardFunc` and friends # Progressive disclosure Source: https://docs.benchflow.ai/progressive-disclosure # Progressive disclosure ## TL;DR `BaseUser` is a Python callback that drives a benchflow rollout across multiple rounds. Each round: the callback sees the previous verifier result and decides what to tell the agent next, or stops the loop. No second LLM, no outbox protocol — just a function that knows how to grade and hint. It was built for the SWE-bench Pro progressive-disclosure use case: the dataset's instructions are long structured specs that overwhelm agents in a single turn. A `BaseUser` lets you compress the spec for round 0, watch which tests fail, then disclose hints from the spec on subsequent rounds — all driven by deterministic Python, not by another LLM acting as a "user." Other agent-eval frameworks model this with a "simulated user" — a second LLM running in a sidecar container that talks to the agent over a side channel. benchflow's `BaseUser` is just in-process Python: no second LLM, no sidecar, no outbox protocol. ```python theme={null} import benchflow as bf from benchflow import FunctionUser, RoundResult from benchflow.rollout import RolloutConfig, Scene from benchflow._utils.benchmark_repos import resolve_source def progressive(round: int, instruction: str, rr: RoundResult | None) -> str | None: if round == 0: return instruction.split("\n")[0] # terse: first line only if rr and (rr.rewards or {}).get("reward", 0) >= 1.0: return None # passed, stop if round >= 3: return None # cap at 3 rounds return ( f"Tests failed:\n{rr.verifier_output}\n\n" # show failures + spec f"Full spec:\n{instruction}" ) config = RolloutConfig( task_path=resolve_source("benchflow-ai/swebenchpro", path="instance_flipt-io__flipt-..."), scenes=[Scene.single(agent="opencode", model="anthropic/claude-sonnet-4-6")], user=FunctionUser(progressive), max_user_rounds=3, environment="daytona", ) result = await bf.run(config) ``` *** ## Case study: SWE-bench Pro SWE-bench Pro tasks ship long, structured `instruction.md` specs (typically 2-5KB) describing API requirements, test fixtures, and expected behaviors. Single-shot agents either drown in the spec or under-engineer because they bail before reading to the bottom. The SWE-bench Pro eval that motivated this feature wanted exactly this loop: ``` round 0 "Fix the bug described here: " agent attempts → tests fail round 1 "Tests failed. Here is the full requirements section: ." agent retries → tests still fail round 2 "Still failing. Here's the full original spec: " agent makes final attempt ``` Rule-based, deterministic, and the "user" never needs to think — the disclosure schedule is fixed. Spinning up a second LLM to play the user role would (a) cost double, (b) introduce nondeterminism, and (c) require an outbox protocol the agent has to learn. ### Validation (2026-04-25, 5 SWE-bench Pro tasks, Daytona, Gemini 3.1 Pro Preview) | Task | Oracle | Single-round baseline | 3-round progressive (final) | Per-round soft-verify | | ----------- | -------------------------------------- | ------------------------------- | --------------------------- | --------------------- | | ansible | ✅ 1.0 | ✅ 1.0 (23 tools, 207s) | ✅ 1.0 (126 tools, 3 rounds) | 0.0 / 0.0 / 0.0 | | flipt | ✅ 1.0 | ❌ 0.0 (61 tools, 1444s) | ❌ 0.0 (195 tools, 3 rounds) | 0.0 / 0.0 / 0.0 | | openlibrary | ✅ 1.0 | ✅ 1.0 (32 tools, 340s) | ✅ 1.0 (82 tools, 3 rounds) | 0.0 / 0.0 / 0.0 | | navidrome | ✅ 1.0 | (not tested) | ❌ 0.0 (145 tools, 3 rounds) | 0.0 / 0.0 / 0.0 | | qutebrowser | ✅ 1.0 (with `cleanup_conftests=false`) | ❌ 0.0 (verifier broken pre-fix) | ✅ 1.0 (183 tools, 3 rounds) | 0.0 / 0.0 / 0.0 | What this run shows and doesn't show: * **The infrastructure works on real SWE-bench Pro tasks.** All 5 tasks completed 3 rounds end-to-end (after one retry on ansible/qutebrowser to clear intermittent flake). Round trajectories captured, soft\_verify runs between rounds, BaseUser callback drives the loop. * **3/5 hit the canonical reward** (ansible, openlibrary, qutebrowser). flipt and navidrome stayed at 0.0 across all three rounds — Gemini 3.1 Pro doesn't crack them with this hint schedule, and progressive disclosure didn't help. * **Per-round soft-verify scored 0.0 even on tasks where the final hardened verify scored 1.0.** Soft-verify runs between rounds without the full hardening sequence (no workspace restore, no process kill so the sandbox stays alive), so its scoring can diverge from the final verifier. The user's hint schedule reacts to soft-verify, not the canonical reward — something to keep in mind when designing the loop. * **First-run flake.** ansible's first run hit a transport EOF after 17min and qutebrowser timed out at 50min. Both succeeded on retry. v0.3.3 adds `agent_idle_timeout` (default 600s) and clearer EOF diagnostics so the next time a hang happens the failure is fast and actionable rather than silent. This is one model on one day, not a published comparison. The notebook at [`examples/swebench_pro_progressive_disclosure.ipynb`](./examples/swebench_pro_progressive_disclosure.ipynb) has the executable cells. *** ## Where it lives in the rollout lifecycle `BaseUser` plugs into the existing `Rollout` lifecycle ([concepts](./concepts.md#rollout-lifecycle)) without changing any of the existing phases. When `RolloutConfig.user` is set, `Rollout._run_user_loop()` replaces the single-pass `connect → execute → disconnect` block with a per-round version: ``` setup() → start() → install_agent() ↓ [oracle setup if oracle_access=True: read /solution, hide it from agent] ↓ user.setup(instruction, solution) ← once ↓ ┌─ user.run(round, instruction, rr) → str | None │ │ None: break │ ↓ │ connect_as(role) │ execute(prompts=[prompt]) │ disconnect() │ ↓ │ soft_verify() ← partial hardening, sandbox stays alive │ ↓ │ build RoundResult, log, repeat └─ │ ↓ (loop ends when user returns None or max_user_rounds reached) [oracle restore: mv /solution_oracle_backup → /solution for final verify] ↓ verify() ← full hardening, final reward ↓ cleanup() ``` Multi-scene / multi-role configs are not compatible with `User` — the loop assumes one Scene with one Role. Setting both raises `ValueError`. *** ## Soft-verify and full-verify: two different verifiers Between rounds, BenchFlow needs to score the agent's progress so the user can react. But the final, end-of-rollout verifier does destructive things (kills the agent, restores the workspace, chowns to root) that would prevent the next round from running. So BenchFlow executes **two** verifier passes: | | Soft-verify (between rounds) | Full-verify (end of rollout) | | --------------------------------------------------------------- | -------------------------------------- | ---------------------------- | | Kills agent processes | ❌ no | ✅ yes | | Restores workspace from snapshot | ❌ no | ✅ optional, task-driven | | Purges agent-injected `conftest.py`, `sitecustomize.py`, `.pth` | ✅ yes | ✅ yes | | Locks down PATH/PYTHONPATH | ✅ yes | ✅ yes | | `chmod 777 /logs/verifier` | ✅ yes (so non-root verifier can write) | n/a (root) | | Runs verifier | ✅ yes | ✅ yes | | Result | feeds `RoundResult.rewards` | the rollout's final score | Soft-verify is intentionally weaker than full-verify — losing some score-gaming protection in exchange for keeping the sandbox alive. The cleanup step still purges agent-injected hook files (`CLEANUP_CMD`), so an agent can't plant a `conftest.py` that flips the round score. *** ## API ### `BaseUser` ```python theme={null} from benchflow import BaseUser, RoundResult class MyUser(BaseUser): async def setup(self, instruction: str, solution: str | None = None) -> None: """Called once before round 0. instruction — the original task instruction (from instruction.md) solution — gold answer if oracle_access=True, else None """ self.spec = instruction self.gold = solution async def run( self, round: int, instruction: str, round_result: RoundResult | None = None, ) -> str | None: """Return the next prompt, or None to stop. round — 0-indexed instruction — original task instruction (unchanged each round) round_result — None on round 0; previous round's outcome on subsequent rounds """ ... ``` ### `RoundResult` Dataclass passed to `run()` from round 1 onward. ```python theme={null} @dataclass class RoundResult: round: int # 0-indexed trajectory: list[dict] # ACP events from this round only rewards: dict | None # verifier rewards (None if verifier crashed) verifier_output: str | None # raw verifier stdout/log verifier_error: str | None # exception message if verifier failed n_tool_calls: int # tool calls in this round ``` ### `PassthroughUser` Sends the instruction unchanged on round 0, stops on round 1. Use it as the explicit single-round-equivalent. ### `FunctionUser` Wraps a plain function as a `BaseUser`. Sync or async — uses `inspect.isawaitable` to detect. ```python theme={null} def fn(round, instruction, rr): ... user = FunctionUser(fn) async def afn(round, instruction, rr): ... user = FunctionUser(afn) ``` ### `RolloutConfig` fields ```python theme={null} user: BaseUser | None = None # the callback max_user_rounds: int = 5 # cap on rounds (loop also stops when user returns None) oracle_access: bool = False # expose gold solution to user.setup() ``` *** ## Oracle access When `oracle_access=True`: 1. Before round 0, the rollout reads `/solution/solve.sh` and passes its contents to `user.setup(instruction, solution=...)`. 2. The rollout moves `/solution` → `/solution_oracle_backup` so the agent can't read it during its rounds. 3. Between rounds, soft-verify temporarily restores `/solution` (some verifiers consult it) then re-hides it. 4. Before the final `verify()`, the rollout permanently restores `/solution`. Step 4 is wrapped in `try/finally` against the user loop: if a round throws, the restore still runs. > ⚠️ Setting `oracle_access=True` *without* a `User` is a misconfiguration — the solution stays exposed to the agent for the entire rollout. benchflow logs a `WARNING` at setup time when this happens. Use cases for oracle access: * **Dataset generation** — the user has the answer, generates an optimal prompt for the agent * **Curriculum learning** — progressively reveal pieces of the solution * **Research** — measure how much oracle information is required for an agent to succeed *** ## Per-task hardening opt-outs The verifier's pre-run cleanup deletes `conftest.py` outside `/tests/` to prevent reward-hacking. Some tasks (qutebrowser) ship legitimate `conftest.py` files that fix real circular imports — deleting them breaks pytest collection. Tasks opt out in `task.toml`: ```toml theme={null} [verifier.hardening] cleanup_conftests = false ``` | Flag | Default | Effect when `false` | | ------------------- | ------- | ---------------------------------------------------------- | | `cleanup_conftests` | `true` | Don't delete `conftest.py` outside `/tests/` before verify | `sitecustomize.py`, `.pth` files, and `*.py` in `/tmp` always get cleaned — they have no legitimate use in a test artifact and disabling them broadens the attack surface beyond what real-world tasks need. Unknown keys in `[verifier.hardening]` are warned and ignored. String values for boolean flags are rejected. *** ## Failure modes The user loop catches exceptions from `user.run()` and stops, with the exception message stored in `Rollout._error`: ``` [User] round 2: prompt='Try again, focusing on...' ERROR user.run() failed at round 2: KeyError: 'spec_section' ``` `soft_verify()` between rounds catches its own timeouts and crashes — they surface as `RoundResult.verifier_error`, not as a rollout-level failure. The next round still runs and the user can decide what to do. Trajectory and tool counts are sliced per round from `Rollout._trajectory`. The session counters reset on `disconnect()`, so each round's `RoundResult.trajectory` and `n_tool_calls` reflect only that round's events, not cumulative. *** ## Comparison with multi-agent simulated user benchflow has two patterns for multi-round agent runs. Neither requires a sidecar container. | Pattern | What "user" is | When to use | | ------------------------------------------------------------------------------------------------------------ | ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **`BaseUser` callback (this doc)** | Python function in the scheduler process | Programmatic, deterministic, rule-based. No second LLM. Cheap. Best for progressive disclosure, curriculum, scripted hints. | | **Multi-role Scene with simulated-user role** ([use-cases §1](./use-cases.md#1-interactive-user-simulation)) | Another LLM with full tool access | Open-ended, conversational. The "user" can read files, check outputs, give nuanced feedback. Best when the user's behavior must itself be adaptive or LLM-quality. | The two coexist. Choose based on whether your "user" needs to think (Scene-based) or just decide (`BaseUser`). For the SWE-bench Pro use case, the disclosure schedule is fixed, the grading is the verifier, and there's nothing for a second LLM to add — `BaseUser` wins on cost and determinism. *** ## Worked examples * [`examples/swebench_pro_progressive_disclosure.ipynb`](./examples/swebench_pro_progressive_disclosure.ipynb) — the SWE-bench Pro case study, executable end-to-end with the latest oracle/baseline data. * [`examples/swebench_pro_user_dogfood.py`](./examples/swebench_pro_user_dogfood.py) — runnable script for any of the 5 SWE-bench Pro tasks. `--task flipt --max-rounds 3`. * [`examples/user_dogfood.py`](./examples/user_dogfood.py) — minimal edit-pdf task with `FunctionUser`, useful as a starting template. # Cli Source: https://docs.benchflow.ai/reference/cli # CLI reference BenchFlow uses a resource-verb pattern: `bench `. ```bash theme={null} bench --version ``` *** ## bench agent > **`bench agent` is agent management only.** `bench agent list` and `bench > agent show` operate on **registered AI agents** (Claude Code, Gemini CLI, > Codex, OpenHands, …) — the programs that solve tasks. Onboarding a third-party > benchmark (scaffold → drive → parity-gate a `benchmarks//` adoption) is a > separate workflow under [`bench eval adopt`](#bench-eval-adopt) (`init` → `convert` → > `verify`). The legacy `bench agent create|run|verify` still work as hidden > deprecated aliases through 0.6, printing a one-line notice; they are removed in > 0.7. ### bench agent list List all registered agents with their protocol and native/default auth requirements. Provider-prefixed models may use provider-specific credentials; Azure Foundry models use `AZURE_API_KEY` plus `AZURE_API_ENDPOINT`. ```bash theme={null} bench agent list ``` ### bench agent show Show details for a specific agent, including native/default auth and a note about provider-specific credentials. ```bash theme={null} bench agent show gemini ``` ## bench eval adopt Bring a third-party benchmark into the environment framework: scaffold a `benchmarks//` package, drive the codex `CONVERT.md` conversion, then parity-gate it (`init` → `convert` → `verify`). These commands were previously `bench agent create|run|verify`, which still work as hidden deprecated aliases through 0.6 (they print a one-line notice and are removed in 0.7). See [Benchmark adoption](../benchmark-adoption.md) for the full walkthrough. ### bench eval adopt init Scaffold `benchmarks//` for a new benchmark adoption. The layout mirrors the reference benchmark `benchmarks/programbench/` and the contract in [`benchmarks/CONVERT.md`](../../benchmarks/CONVERT.md): it writes `benchflow.py` (converter), `main.py`, `parity_test.py`, `run_.py`, `.yaml`, `benchmark.yaml`, `parity_experiment.json` (status `template`), `README.md`, and `__init__.py`. It is fail-closed: the slug is validated (lowercase, leading letter, single internal hyphens, max 64 chars) and the command refuses to overwrite an existing benchmark directory. ```bash theme={null} bench eval adopt init my-bench bench eval adopt init my-bench --benchmarks-dir ./benchmarks ``` | Flag | Default | Description | | ------------------ | ------------------ | ---------------------------- | | `--benchmarks-dir` | repo `benchmarks/` | Target benchmarks/ directory | ### bench eval adopt convert Drive the `CONVERT.md` adoption workflow by launching the host `codex` CLI. The command assembles the adoption context (the source, the target `benchmarks//` path, the adoption skills, and the embedded `benchmarks/CONVERT.md` guide) and runs `codex exec` against the repo root to drive the conversion toward a `benchmarks//` pull request. It is fail-closed on credentials: `codex` needs `OPENAI_API_KEY` (or `CODEX_API_KEY`) in the environment, or a `~/.codex/auth.json` from `codex login`, otherwise the command exits before assembling any context. Use `--dry-run` to print the exact launch command without running it (no credentials required). When `--name` is omitted the slug is derived from the source basename. ```bash theme={null} # Print the codex launch command without running it bench eval adopt convert https://github.com/org/some-benchmark --dry-run # Launch the host codex driver against a local source bench eval adopt convert ./vendor/some-benchmark --name my-bench --model o3 ``` | Flag | Default | Description | | ---------------------- | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `--name` | derived from source | Benchmark slug (default: from source basename) | | `--model` | codex default | Model for the codex driver | | `--dry-run` | `false` | Print the launch command, do not run | | `--codex-bin` | `codex` | Host codex binary | | `-c`, `--codex-config` | — | Codex config override as `key=value`, passed through to codex as `-c key=value`; repeatable. Use it to work around host `~/.codex/config.toml` drift without editing the file — e.g. `-c service_tier=flex` when an installed codex version rejects a stale value. | ### bench eval adopt verify Run the parity gate for an adopted benchmark and emit a confidence verdict. It reads `benchmarks//parity_experiment.json` and scores two layers: a deterministic conversion-faithfulness floor (every compared criterion's converted verdict must match the original's verdict on identical inputs) and a statistical reward-distribution layer (every legacy-vs-converted reward delta must sit within `--tolerance`). The gate is parity-only — a faithful conversion reproduces the original's behavior, including any reward-hackability the source has; it never "improves" or sanitizes the source. The verdict is one of `parity-confirmed`, `parity-divergent`, or `insufficient-evidence` (no recorded comparisons). On any non-confirmed verdict the command exits non-zero and emits a draft GitHub issue body for human support — printed to stdout, or written to `--issue-out`. The draft is never filed automatically. Pass `--roundtrip-task` to also run the structural round-trip conformance check on a concrete task directory. By default the gate **scores the recorded** `parity_experiment.json` — fast, but it trusts an artifact the conversion produced about itself. Pass `--rerun` to **independently re-execute** `parity_test.py --mode side-by-side` and score its fresh output instead. `--rerun` is fail-closed: a missing/failing `parity_test.py`, a timeout, or output that is not in the scoreable `parity_experiment.json` shape all exit non-zero (rather than silently reporting `insufficient-evidence`). ```bash theme={null} bench eval adopt verify my-bench bench eval adopt verify my-bench --tolerance 0.05 --issue-out divergence.md bench eval adopt verify my-bench --roundtrip-task benchmarks/my-bench/tasks/example bench eval adopt verify my-bench --rerun # re-run parity_test.py, score fresh output ``` | Flag | Default | Description | | ------------------ | ------------------ | --------------------------------------------------------------------------------------------------------------------------- | | `--benchmarks-dir` | repo `benchmarks/` | Target benchmarks/ directory | | `--tolerance` | `0.02` | Max abs reward delta (statistical layer) | | `--issue-out` | — | Write the divergence issue draft to this path instead of stdout | | `--roundtrip-task` | — | Also run the structural round-trip check on this task dir | | `--rerun` | `false` | Re-execute `parity_test.py --mode side-by-side` and score its fresh output instead of the recorded `parity_experiment.json` | ## bench eval ### bench eval create Create and run an evaluation. Use it for YAML configs and batch runs; it also accepts a single task directory. ```bash theme={null} # From YAML config bench eval create --config benchmarks/harvey-lab/harvey-lab-gemini-flash-lite.yaml # From remote repo (fast Daytona batch; token usage may be unavailable) bench eval create \ --source-repo benchflow-ai/skillsbench \ --source-path tasks \ --agent gemini \ --model gemini-3.1-flash-lite-preview \ --sandbox daytona \ --concurrency 64 \ --sandbox-setup-timeout 300 # From remote repo with required token usage telemetry bench eval create \ --source-repo benchflow-ai/skillsbench \ --source-path tasks \ --agent gemini \ --model gemini-3.1-flash-lite-preview \ --sandbox daytona \ --usage-tracking required \ --concurrency 16 \ --sandbox-setup-timeout 300 # From local directory bench eval create --tasks-dir ./tasks --agent gemini --model gemini-3.1-flash-lite-preview # From a hosted PrimeIntellect / Verifiers environment bench eval create \ --source-env primeintellect/general-agent \ --source-env-version 0.1.1 \ --source-env-arg task=calendar_scheduling_t0 \ --agent gemini \ --model google/gemini-2.5-flash-lite # Single task with mounted skills and the recommended skill nudge bench eval create \ --tasks-dir tasks/pdf-fix \ --agent gemini \ --model gemini-3.1-flash-lite-preview \ --sandbox daytona \ --skill-mode with-skill \ --agent-env BENCHFLOW_SKILL_NUDGE=name # Pinned registry dataset: resolves skillsbench@1.1, verifies task digests, # and stamps dataset identity into every result.json/config.json bench eval create -d skillsbench@1.1 --agent gemini --model gemini-3.1-flash-lite-preview ``` | Flag | Default | Description | | ----------------------------------- | -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `--config` | — | YAML config file | | `--tasks-dir` | — | Local task dir (single task with task.toml, or parent of many) | | `-d`, `--dataset` | — | Registry dataset to run as `@` (e.g. `skillsbench@1.1`). Resolves the pinned snapshot from the registry, clones tasks at their pinned commit, verifies each task's sha256 content digest, and checks the dataset's `bench_version` range against the installed benchflow. Each `result.json`/`config.json` is stamped with `dataset_name`, `dataset_version`, and the task's `task_digest`. | | `--registry` | skillsbench registry | Dataset registry JSON URL or local file. Only valid with `--dataset`. | | `--source-repo` | — | Remote repo as `org/repo` (e.g. `benchflow-ai/skillsbench`) | | `--source-path` | — | Subpath within the repo (e.g. `tasks`) | | `--source-ref` | — | Branch or tag to clone (e.g. `main`) | | `--source-env` | — | Hosted environment source (e.g. `primeintellect/general-agent`) | | `--source-env-version` | — | Hosted environment version | | `--source-env-arg` | — | Hosted environment argument as `KEY=VALUE`; repeatable | | `--source-env-num-examples` | `1` | Number of hosted environment examples | | `--source-env-rollouts-per-example` | `1` | Rollouts per hosted environment example | | `--source-env-max-tokens` | `1024` | Max tokens for hosted environment model calls | | `--source-env-temperature` | `0.0` | Temperature for hosted environment model calls | | `--source-env-sampling-arg` | — | Verifiers sampling argument as `KEY=VALUE`; repeatable (for example `reasoning_effort=minimal`) | | `--agent` | `claude-agent-acp` | Agent name | | `--model` | Agent default | Model ID | | `--reasoning-effort` | — | Agent reasoning/thinking effort when the agent exposes one (e.g. `max`) | | `--sandbox` | `docker` | Sandbox: docker, daytona, or modal | | `--usage-tracking` | `auto` | Token usage telemetry policy: `auto`, `required`, or `off` | | `--environment-manifest` | — | Path to an Environment-plane manifest (`environment.toml`); applied to every rollout in the batch | | `--prompt` | `instruction.md` | Prompt to send to the agent; repeatable for multi-prompt runs | | `--concurrency` | `4` | Max concurrent tasks (batch mode only) | | `--build-concurrency` | `--concurrency` | Max concurrent docker image builds; set lower (e.g. `8`) when `--concurrency` is high to avoid overwhelming the docker daemon | | `--worker-concurrency` | — | Run batch eval through isolated worker subprocesses, each with at most this many concurrent tasks; `--concurrency` remains the aggregate target | | `--worker-retries` | `1` | Retry a crashed worker shard this many times, resuming its jobs dir | | `--worker-start-stagger-sec` | `1.0` | Seconds to stagger worker starts to avoid Daytona connection storms | | `--agent-idle-timeout` | (built-in default) | Abort ACP prompts after this many idle seconds; `0` disables idle detection | | `--jobs-dir` | `jobs` | Output directory | | `--sandbox-user` | `agent` | Sandbox user (null for root) | | `--sandbox-setup-timeout` | `120` | Timeout in seconds for sandbox user setup | | `--skills-dir` | — | Advanced custom skills directory; valid only with `--skill-mode with-skill`. Omit it to use each task's `environment/skills`. | | `--skill-mode` | `no-skill` | Skill mode: `no-skill`, `with-skill`, or `self-gen` | | `--skill-creator-dir` | — | Path to a `skill-creator` directory (or a skills root containing it); used when `--skill-mode self-gen` | | `--self-gen-no-internet` | `false` | Disable web tools for the self-generated skill run | | `--agent-env` | — | Agent environment variable as `KEY=VALUE`; repeatable | | `--include` | — | Only run these task names; repeatable (e.g. `--include jax-computing-basics --include data-to-d3`) | | `--exclude` | — | Skip these task names; repeatable (e.g. `--exclude quantum-numerical-simulation`) | | `--loop-strategy` | — | Wrap each rollout in a loop, e.g. `verify-retry:k=3,feedback=names` or `self-review:k=3` (omit for single-shot) | | `--ignore-bench-version` | `false` | With `--dataset`, skip the dataset's `bench_version` compatibility gate | When mounting skills, the recommended docs default is `--agent-env BENCHFLOW_SKILL_NUDGE=name`. See [Architecture: skill loading](../architecture.md#skill-loading) for how `with-skill` mode is registered with each agent and how the nudge modes differ. Daytona batch runs collect provider token/cost telemetry by default with a sandbox-local LiteLLM gateway. Use `--usage-tracking required` when missing telemetry should fail the rollout, or `--usage-tracking off` for recovery runs that should leave provider traffic untouched. `--source-env` is for external hosted environment hubs. The first supported runner is PrimeIntellect / Verifiers: BenchFlow preserves the hosted identity (`env_uid`, `hub_url`), installs the versioned package into an isolated local virtual environment, and runs `vf-eval`. `--sandbox` remains the BenchFlow task sandbox selector for local/repo task sources; Verifiers source environments own their own harness and sandbox behavior. `--model` is passed to the Verifiers model endpoint; use a model id available to that provider. Provider-specific sampling options are not inferred; pass them explicitly with `--source-env-sampling-arg`. ### bench eval list List completed evaluations from a jobs directory. ```bash theme={null} bench eval list jobs/ ``` ### bench eval metrics Collect and display metrics (pass/fail/score, memory score, tool calls, duration) from a jobs directory. Use `--json` for machine-readable output. ```bash theme={null} bench eval metrics jobs/ bench eval metrics jobs/ --json ``` ### bench eval view Serve a trial trajectory viewer in the browser for a rollout or job directory. ```bash theme={null} bench eval view jobs/run/task__abc123 bench eval view jobs/ --port 9000 ``` ## bench skills ### bench skills list List skills discovered under the default skills roots (or `--dir`). ```bash theme={null} bench skills list bench skills list --dir ./skills ``` ### bench skills eval Evaluate a skill against its evals.json test cases. ```bash theme={null} bench skills eval skills/my-skill/ \ --agent gemini \ --model gemini-3.1-flash-lite-preview \ --sandbox daytona ``` *** ## bench tasks ### bench tasks init Scaffold a new benchmark task. ```bash theme={null} bench tasks init my-new-task bench tasks init my-new-task --dir tasks/ bench tasks init my-new-task --format legacy ``` | Flag | Default | Description | | ---------- | --------- | --------------------------------------------------------------------------------------------------------- | | `--format` | `task-md` | Task format: `task-md` (native single-document) or `legacy` (split `task.toml` + `instruction.md` layout) | ### bench tasks check Validate a task directory (`task.md` or legacy `task.toml` + `instruction.md`, `environment/Dockerfile`, `verifier/` or legacy `tests/`). ```bash theme={null} bench tasks check tasks/my-task ``` With `--level`, validation runs at a chosen depth: `schema`, `structural`, `runtime-capability`, `publication-grade`, `acceptance`, or `acceptance-live`. Acceptance-level errors such as `acceptance validation requires benchflow.evidence mapping` refer to the `benchflow.evidence` schema documented in the "Assets, Provenance, And Evidence" section of `docs/task-standard.md`. ### bench tasks migrate Convert a legacy `task.toml` + `instruction.md` task into the unified `task.md` format. By default the legacy files are kept alongside the new `task.md`. ```bash theme={null} bench tasks migrate tasks/my-task bench tasks migrate tasks/my-task --overwrite --remove-legacy ``` | Flag | Default | Description | | ----------------- | ------- | ------------------------------------------------------------------------------- | | `--overwrite` | `false` | Replace an existing task.md | | `--remove-legacy` | `false` | Delete split files and promote tests/solution aliases after task.md is verified | ### bench tasks normalize Expand minimal `task.md` authoring profiles into the canonical `task.md` form. Prints the normalized document to stdout unless told otherwise. ```bash theme={null} bench tasks normalize tasks/my-task bench tasks normalize tasks/my-task --write bench tasks normalize tasks/my-task -o normalized-task.md ``` | Flag | Default | Description | | ---------------- | ------- | ----------------------------------------------------------- | | `--output`, `-o` | — | Write normalized task.md to this path instead of stdout | | `--write` | `false` | Replace task.md in place with the normalized canonical form | ### bench tasks export Export a `task.md` task to a Harbor/Pier-compatible split layout, with a compatibility loss report written to `compatibility/export-report.json` in the export directory. ```bash theme={null} bench tasks export tasks/my-task out/my-task-split bench tasks export tasks/my-task --report-only bench tasks export tasks/my-task out/my-task-split --target pier --overwrite ``` Arguments: `TASK_DIR` (task directory to export) and optional `OUTPUT_DIR` (destination split-layout directory; may be omitted with `--report-only`). | Flag | Default | Description | | --------------- | -------- | --------------------------------------------------------- | | `--target` | `harbor` | Compatibility target: `harbor` or `pier` | | `--overwrite` | `false` | Replace an existing export directory | | `--report-only` | `false` | Print the compatibility loss report without writing files | ### bench tasks digest Compute the content digest that pins a task's files, independent of git — the sha256 the dataset registry keys on (matches the digests `bench eval create -d` verifies and the `task_digest` stamped into every `result.json`). Recognizes both legacy `task.toml` tasks and native `task.md` tasks. Given a single task directory it prints the digest; given a directory of tasks it prints one ` ` line per task. Output goes to stdout via `echo` (not Rich), so it is safe to pipe into machine-readable tooling. ```bash theme={null} bench tasks digest tasks/my-task # -> sha256: bench tasks digest tasks/ # one " sha256:" line per task ``` Arguments: `PATH` (a task directory, or a directory of task directories). ### bench tasks generate Generate benchmark task directories from real agent traces. ```bash theme={null} bench tasks generate --from-local --project my-repo --limit 5 bench tasks generate --from-file session.jsonl --dry-run bench tasks generate --from-hf opentraces-test --limit 50 ``` | Flag | Default | Description | | ---------------- | --------------------- | ---------------------------------------------------- | | `--from-local` | — | Generate from local Claude Code sessions | | `--from-file` | — | Generate from a JSONL trace file | | `--from-hf` | — | Generate from a HuggingFace dataset ID or alias | | `--output` | `tasks` | Output directory for generated tasks | | `--projects-dir` | `~/.claude/projects/` | Claude Code projects directory | | `--project` | — | Filter local sessions by project path substring | | `--format` | `auto` | Trace format override | | `--split` | `train` | HuggingFace dataset split | | `--max-rows` | `100` | Max rows to download from HuggingFace | | `--limit` | `20` | Max traces to process | | `--min-steps` | `2` | Minimum steps per trace | | `--outcome` | — | Filter by outcome: success, failure, unknown | | `--author` | `benchflow-traces` | Author name for generated task metadata | | `--task-format` | `task-md` | Generated task package format: `task-md` or `legacy` | | `--dry-run` | `false` | Preview traces without generating tasks | ### bench tasks list-sources List known HuggingFace trace datasets. The aliases listed here can be passed to `bench tasks generate --from-hf`. ```bash theme={null} bench tasks list-sources ``` ## bench sandbox Local sandbox lifecycle: provision a task on a docker/daytona/modal backend, list active sandboxes, and reap stale ones. ### bench sandbox create Create an environment object from a task directory. This validates environment construction but does not start the sandbox. ```bash theme={null} bench sandbox create tasks/my-task --sandbox daytona ``` ### bench sandbox list List active local (Daytona) sandboxes. ```bash theme={null} bench sandbox list ``` ### bench sandbox cleanup Clean up orphaned Daytona sandboxes. By default this deletes sandboxes older than 24 hours; use `--dry-run` to preview what would be deleted. ```bash theme={null} bench sandbox cleanup --dry-run --max-age 1440 ``` Daytona-backed evals also reap orphaned sandboxes automatically at run start (failure states such as `BUILD_FAILED` are reaped sooner than healthy ones, and an idle-activity guard means concurrent live runs are never reaped). Set `BENCHFLOW_DAYTONA_AUTO_REAP` to any of `0`/`false`/`no`/`off` (case-insensitive) to disable that automatic pass and rely on the manual command above. ## bench environment (deprecated) `bench environment` is a hidden **deprecated alias group**, removed in 0.7. The local lifecycle moved to [`bench sandbox`](#bench-sandbox) (`create`/`list`/`cleanup`) and hosted-provider browsing to [`bench hub env`](#bench-hub-env). The old `bench environment create|list|cleanup` and `show|inspect` (plus `list --provider`/`--hub`) still work, each printing a one-line stderr notice. ## bench hub External environment hubs: compatibility checks (`check`) and browsing a hosted provider's environments (`env`). ### bench hub env Read-only browsing of a hosted provider's environments (PrimeIntellect "Environments"). To *run* one, use [`bench eval create --source-env`](#bench-eval-create). ```bash theme={null} bench hub env list --provider primeintellect --owner primeintellect --search general-agent --limit 5 bench hub env show primeintellect/general-agent --version 0.1.1 bench hub env inspect primeintellect/general-agent --version 0.1.1 --path README.md ``` ### bench hub check Inventory or structurally check representative tasks from an environment hub's registry. Defaults to an inventory pass against the public Harbor registry JSON. ```bash theme={null} # Inventory the public Harbor hub registry bench hub check # Structural check, two tasks per dataset, JSONL output bench hub check --level check --tasks-per-dataset 2 --out hub.jsonl ``` | Flag | Default | Description | | --------------------- | -------------------------- | ------------------------------------------- | | `--registry` | Harbor public registry URL | Harbor registry JSON URL or local file | | `--tasks-per-dataset` | `2` | Representative tasks selected per dataset | | `--level` | `inventory` | Compatibility level: `inventory` or `check` | | `--out` | — | Optional JSONL output path | | `--cache-dir` | `.cache/hub/harbor` | Cache directory for sparse clones | | `--limit` | — | Optional cap on selected task refs | ## YAML Config Format ### Batch config with skills and skill nudge ```yaml theme={null} source: repo: benchflow-ai/skillsbench path: tasks environment: daytona concurrency: 64 sandbox_setup_timeout: 300 agent: gemini model: gemini-3.1-flash-lite-preview skill_mode: with-skill skills_dir: shared-skills/ agent_env: BENCHFLOW_SKILL_NUDGE: name max_retries: 2 ``` ### Multi-scene (BYOS skill generation) Use the Python API for multi-scene experiments. `bench eval create --config` is for batch job configs; scene configs are loaded with `benchflow._utils.yaml_loader` or built directly in Python. ```yaml theme={null} task_dir: tasks/my-task environment: daytona sandbox_setup_timeout: 300 scenes: - name: skill-gen roles: - name: creator agent: gemini model: gemini-3.1-flash-lite-preview turns: - role: creator prompt: "Analyze the task and write a skill document to /app/generated-skill.md" - name: solve roles: - name: solver agent: gemini model: gemini-3.1-flash-lite-preview turns: - role: solver ``` *** ## bench continue Resume a previous, unfinished (timed-out) `openhands` run to completion via record-replay. Standalone — it does not touch the normal run path. See [Continuing timed-out runs](../continue-runs.md) for the full guide. ```bash theme={null} bench continue path/to/original/run-folder --tasks-dir path/to/tasks ``` Key options: `--model` (override the live-continuation model; defaults to the original run's model), `--timeout`, `--output`, `--require-timeout`, `--strict-divergence`, `--replay-only` (rebuild via replay and stop at the cut-point — no live model or API key needed), and `--proxy-mode` (replay proxy placement: `auto`, `host`, or `sandbox`; default `auto` uses sandbox-local replay for Daytona/Modal and host replay for Docker). ### bench continue-batch Continue all timed-out OpenHands runs found under a directory tree. Discovers run folders (`config.json` + `trajectory/llm_trajectory.jsonl`) recursively, continues each, and prints a JSON batch summary (exits 1 if any continuation failed). ```bash theme={null} bench continue-batch path/to/jobs-root --tasks-dir path/to/tasks ``` | Flag | Default | Description | | --------------------- | -------------------- | ----------------------------------------------------------------------------- | | `--tasks-dir` | — | Directory holding task sources; required unless the recorded task path exists | | `--model` | original run's model | Override the live-continuation model | | `--timeout` | — | Wall-clock budget per continuation | | `--output` | — | Output jobs dir for continued runs | | `--concurrency` | `100` | Maximum number of continuation runs in flight | | `--limit` | — | Limit discovered timeout folders | | `--strict-divergence` | `false` | Abort a run if replay leaves the original rails | | `--proxy-mode` | `auto` | Replay proxy placement: `auto`, `host`, or `sandbox` | # Python api Source: https://docs.benchflow.ai/reference/python-api # Python API The Rollout/Scene API is the primary way to run agent benchmarks programmatically. ## Install ```bash theme={null} uv tool install --prerelease allow 'benchflow==0.6.0' ``` ## Quick Start ```python theme={null} import asyncio import benchflow as bf result = asyncio.run(bf.run("gemini", task_path="tasks/my-task", model="gemini-3.1-flash-lite-preview")) print(f"Reward: {result.rewards}") print(f"Tool calls: {result.n_tool_calls}") ``` ## Core Types ### RolloutConfig Declarative configuration for a rollout — a sequence of Scenes in a shared sandbox. ```python theme={null} from pathlib import Path from benchflow import RolloutConfig, Scene, Role, Turn # Single-agent (simplest) config = RolloutConfig( task_path=Path("tasks/my-task"), scenes=[Scene.single(agent="gemini", model="gemini-3.1-flash-lite-preview")], environment="daytona", sandbox_setup_timeout=120, ) # Multi-scene BYOS (skill-gen → solve) config = RolloutConfig( task_path=Path("tasks/my-task"), scenes=[ Scene(name="prep", roles=[Role("gen", "gemini", "gemini-3.1-flash-lite-preview")], turns=[Turn("gen", "Generate a skill for this task...")]), Scene(name="solve", roles=[Role("solver", "gemini", "gemini-3.1-flash-lite-preview")], turns=[Turn("solver")]), ], environment="daytona", sandbox_setup_timeout=120, ) ``` Set `sandbox_setup_timeout` when sandbox user setup needs more than the default 120 seconds. The same field is also available on `JobConfig` and `RuntimeConfig`. ### Scene Authoring sugar for role, prompt, and skill attribution. Scenes compile to explicit rollout Steps before execution; there is no runtime Scene object or message scheduler. ```python theme={null} # Single-role shortcut scene = Scene.single(agent="gemini", model="gemini-3.1-flash-lite-preview") # Multi-role with explicit turn order scene = Scene( name="coder-reviewer", roles=[ Role("coder", "gemini", "gemini-3.1-flash-lite-preview"), Role("reviewer", "gemini", "gemini-3.1-flash-lite-preview"), ], turns=[ Turn("coder"), # None prompt = instruction.md Turn("reviewer", "Review the current workspace."), Turn("coder", "Fix the issues."), ], ) ``` ### Rollout The execution engine — decomposed into independently-callable phases. ```python theme={null} from benchflow import Rollout rollout = await Rollout.create(config) # Full lifecycle (most common) result = await rollout.run() # Manual composition (for custom flows) await rollout.setup() await rollout.start() await rollout.install_agent() await rollout.connect() await rollout.execute(prompts=["custom prompt"]) await rollout.disconnect() await rollout.verify() await rollout.cleanup() ``` ### RuntimeConfig Runtime-level configuration for the `Agent + Environment` execution path. ```python theme={null} from benchflow.runtime import Agent, Environment, Runtime, RuntimeConfig config = RuntimeConfig(sandbox_setup_timeout=300) agent = Agent("gemini", model="gemini-3.1-flash-lite-preview") env = Environment.from_task("tasks/X", sandbox="daytona") runtime = Runtime(env, agent, config=config) result = await runtime.execute() ``` ### bf.run() Convenience function — multiple calling conventions: ```python theme={null} import benchflow as bf # 1. RolloutConfig (full control) result = await bf.run(config) # 2. Agent + Environment (0.3 style) agent = bf.Agent("gemini", model="gemini-3.1-flash-lite-preview") env = bf.Environment.from_task("tasks/X", sandbox="daytona") runtime_config = bf.RuntimeConfig(sandbox_setup_timeout=300) result = await bf.run(agent, env, runtime_config) # 3. String shortcut (simplest) result = await bf.run( "gemini", task_path="tasks/X", model="gemini-3.1-flash-lite-preview", config=bf.RuntimeConfig(sandbox_setup_timeout=300), ) ``` ## Rollout Lifecycle ``` Rollout.run() │ ├─ setup() — resolve config, create env object ├─ start() — spin up sandbox, upload task files, start services ├─ install_agent() — install agent binary, credentials, sandbox user │ (sandbox user setup: create non-root user, prepare │ small config/auth dirs, chown the workspace — no │ recursive copy of /root tool trees; agent binaries │ must live on shared prefixes like /usr/local/bin) ├─ compile scenes → Steps ├─ for step in steps: │ ├─ connect_as(role) — open/reuse ACP session for this role │ └─ execute(prompt) — send prompt, collect trajectory, grow tree ├─ verify() — run verifier, collect rewards └─ cleanup() — stop sandbox ``` Key: scene boundaries are gone by execution time; role changes are represented as Step metadata and handled by the rollout executor. ## Multi-Turn vs Multi-Round | Pattern | Roles | Turns | Communication | Example | | --------------- | ----- | ----- | -------------------------------- | ------------------ | | **Single-turn** | 1 | 1 | — | Baseline benchmark | | **Multi-turn** | 1 | 2+ | Same session, sequential prompts | Self-review | | **Multi-role** | 2+ | 2+ | Explicit prompt sequence | Coder + Reviewer | **Multi-turn** = same agent gets multiple prompts. Use when a second pass catches errors (self-review, iterative refinement). The agent keeps its context across turns. **Multi-role** = different agents receive explicit turns. Use when tasks need multiple perspectives (code review, client-advisor). Any handoff text must be part of the declared prompt or agent-native communication, not a BenchFlow Scene scheduler. Both use the same API — `RolloutConfig` with different `Scene` configurations. ## Multi-Agent Patterns ### Coder + Reviewer (followup-bench) ```python theme={null} config = RolloutConfig( task_path=task_path, scenes=[Scene( roles=[Role("coder", "gemini", "flash"), Role("reviewer", "gemini", "flash")], turns=[ Turn("coder"), Turn("reviewer", "Review /app/. Summarize any issues."), Turn("coder", "Read feedback and fix."), ], )], environment="daytona", ) ``` ### Skill Generation + Solve (BYOS) ```python theme={null} config = RolloutConfig( task_path=task_path, scenes=[ Scene(name="skill-gen", roles=[Role("gen", "gemini", "flash")], turns=[Turn("gen", "Generate a skill document to /app/generated-skill.md")]), Scene(name="solve", roles=[Role("solver", "gemini", "flash")], turns=[Turn("solver")]), ], environment="daytona", ) ``` ## User-Driven Loops Use `BaseUser` or `FunctionUser` when one agent should run multiple rounds and Python should decide the next prompt from verifier feedback. This is the progressive-disclosure path: the user callback can stop early, read `RoundResult` after each `soft_verify()`, and optionally receive the oracle solution during `setup()` when `oracle_access=True`. ```python theme={null} from pathlib import Path from benchflow import FunctionUser, RolloutConfig, RoundResult, Scene def user(round: int, instruction: str, rr: RoundResult | None) -> str | None: if round == 0: return instruction.splitlines()[0] if rr and (rr.rewards or {}).get("reward") == 1.0: return None return f"Tests failed:\n{rr.verifier_output}\n\nUse the full spec:\n{instruction}" config = RolloutConfig( task_path=Path("tasks/my-task"), scenes=[Scene.single(agent="gemini", model="gemini-3.1-flash-lite-preview")], user=FunctionUser(user), max_user_rounds=3, environment="daytona", ) result = await bf.run(config) ``` Use multi-role Scenes when another LLM should act as the reviewer or simulated user. Use `BaseUser` when the loop is deterministic or verifier-driven. See [`progressive-disclosure.md`](../progressive-disclosure.md) and [`docs/examples/scene-patterns.ipynb`](../examples/scene-patterns.ipynb). ## YAML Rollout Configs ```python theme={null} from benchflow._utils.yaml_loader import rollout_config_from_yaml config = rollout_config_from_yaml("rollout.yaml") result = await bf.run(config) ``` ## Registered Agents | Agent | Protocol | Auth | Aliases | | ------------------ | -------- | ---------------------------------------------------------------------- | -------- | | `gemini` | ACP | GEMINI\_API\_KEY | — | | `claude-agent-acp` | ACP | ANTHROPIC\_API\_KEY | `claude` | | `codex-acp` | ACP | OPENAI\_API\_KEY, CODEX\_API\_KEY, CODEX\_ACCESS\_TOKEN, or host login | `codex` | | `opencode` | ACP | inferred from model/provider | — | | `openhands` | ACP | LLM\_API\_KEY | `oh` | | `pi-acp` | ACP | ANTHROPIC\_API\_KEY | `pi` | | `openclaw` | ACP | inferred from model | — | The Auth column shows each agent's native/default credentials. Provider-prefixed models can use provider-specific credentials instead; for example, Azure Foundry models use `AZURE_API_KEY` plus `AZURE_API_ENDPOINT` with prefixes such as `azure-foundry-openai/gpt-5.5` or `azure-foundry-anthropic/claude-opus-4-5`. BenchFlow routes these providers through LiteLLM on both Docker and Daytona. Any agent can be prefixed with `acpx/` to run via [ACPX](https://acpx.sh/) (e.g. `acpx/gemini`, `acpx/claude`). ACPX is a headless ACP client with persistent sessions and crash recovery. The underlying agent's install, env, credentials, and skill paths are preserved. ## Retry and Error Handling Rollout.run() catches common errors: * `TimeoutError` — agent exceeded timeout * `ConnectionError` — SSH/ACP pipe closed (retried 3x with exponential backoff) * `ACPError` — agent protocol error Evaluation-level retry with `RetryConfig`: ```python theme={null} from benchflow.evaluation import Evaluation, EvaluationConfig, RetryConfig config = EvaluationConfig( retry=RetryConfig( max_retries=2, wait_multiplier=2.0, min_wait_sec=1.0, max_wait_sec=30.0, ), ) ``` *** ## Sandbox and Reward Types ### Sandbox Protocol The `Sandbox` protocol defines the interface any sandbox backend must implement. Docker and Daytona are built-in; you can bring your own (Modal, Firecracker, E2B, etc.). ```python theme={null} from benchflow import Sandbox, ImageBuilder, ImageConfig, ImageRef # Sandbox is a runtime-checkable Protocol class MySandbox: async def exec(self, cmd: str, *, user: str = "root", timeout_sec: int = 30) -> ExecResult: ... async def upload_file(self, src: Path, dst: str) -> None: ... async def download_file(self, src: str, dst: Path) -> None: ... async def start(self) -> None: ... async def stop(self, *, delete: bool = True) -> None: ... # ... plus snapshot/restore + host/expose_ports; see sandbox/protocol.py assert isinstance(my_sandbox, Sandbox) # works at runtime ``` ### Rubric + RewardFunc (Composable Rewards) Declarative scoring via composable reward functions. ```python theme={null} from benchflow import Rubric, RewardFunc, RewardEvent, VerifyResult from benchflow import TestRewardFunc, StringMatchRewardFunc, LLMJudgeRewardFunc # Built-in reward functions test_reward = TestRewardFunc() # runs pytest, binary pass/fail match_reward = StringMatchRewardFunc(expected="hello world") # Compose into a weighted Rubric rubric = Rubric( reward_funcs=[test_reward, match_reward], weights=[0.7, 0.3], ) # Score a workspace result: VerifyResult = await rubric.score(rollout_dir=my_rollout_dir) print(result.reward) # weighted float [0.0, 1.0] print(result.events) # list[RewardEvent] — per-function breakdown ``` ### Adapters (Inspect AI + ORS) Convert between BenchFlow types and external frameworks. ```python theme={null} from benchflow import InspectAdapter, ORSAdapter, to_inspect_task, to_ors_reward # BenchFlow Scene → Inspect AI task format inspect_task = to_inspect_task(scene, rubric=rubric) # BenchFlow VerifyResult → ORS reward format ors_payload = to_ors_reward(verify_result) ``` ### Evaluation Batch orchestration with concurrency and retries. ```python theme={null} from benchflow import Evaluation, EvaluationConfig, EvaluationResult, RetryConfig # EvaluationConfig holds the per-job settings (agent/model/environment/...) # applied to every task discovered under tasks_dir. config = EvaluationConfig( model="gemini-3.1-flash-lite-preview", environment="daytona", concurrency=8, retry=RetryConfig(max_retries=2), ) evaluation = Evaluation(tasks_dir="tasks", jobs_dir="jobs/my-run", config=config) eval_result: EvaluationResult = await evaluation.run() ``` # Release Source: https://docs.benchflow.ai/release # Release Channels BenchFlow uses two PyPI release channels with the same package name: * **Public** releases are stable versions such as `0.6.0`. They are created by pushing a matching `v` tag. * **Internal preview** releases are development versions such as `0.6.1.dev123`. They are created automatically after the `test` workflow passes for a push to `main`. Current release state: * `0.6.0` is **published on PyPI** (tag `v0.6.0`). Use the public install commands below. * `main` currently carries `0.6.0`. To resume internal-preview builds for the next line, bump `main` to `0.6.1.dev0` — internal previews then publish as `0.6.1.dev` automatically after the `test` workflow passes on `main`. ## Install and Upgrade Commands Use the public channel by default. Opt into internal preview only when you want the newest build from `main` before the next public tag. Public Python package users: ```bash theme={null} pip install --upgrade benchflow ``` Public `uv`-managed CLI users: ```bash theme={null} uv tool install --prerelease allow --upgrade 'benchflow==0.6.0' ``` The exact `benchflow==0.6.0` pin keeps `uv` on the public release while `--prerelease allow` permits the release-candidate LiteLLM dependency used by this package line. Internal preview Python package users: ```bash theme={null} pip install --pre --upgrade benchflow ``` Internal preview `uv`-managed CLI users: ```bash theme={null} uv tool install --prerelease allow --upgrade benchflow ``` The preview CLI command intentionally omits the exact public pin, so `uv` selects the latest `0.6.1.dev` package. If a machine was previously installed with `pip install --user` or another non-`uv tool` method, the command can fail with `Executables already exist: bench, benchflow`. In that case, rerun the same command with `--force`: ```bash theme={null} uv tool install --prerelease allow --upgrade --force benchflow ``` For downstream projects that use `uv`, keep public dependencies pinned unless the project intentionally tracks preview builds: ```bash theme={null} uv add --prerelease allow 'benchflow==0.6.0' uv lock --upgrade-package benchflow --prerelease allow ``` To lock the latest internal preview instead: ```bash theme={null} uv add --prerelease allow benchflow uv lock --upgrade-package benchflow --prerelease allow ``` ## Version Model `pyproject.toml` on `main` should track the next public version as `.dev0`. For example, after publishing `0.6.0`, bump `main` to: ```toml theme={null} version = "0.6.1.dev0" ``` The internal preview workflow rewrites that version only inside the CI build, using the successful `test` workflow run number: ```text theme={null} 0.6.1.dev0 in git -> 0.6.1.dev123 on PyPI ``` This keeps public and internal preview ordering correct: `0.6.1.dev123` is a preview of the future `0.6.1`, while `0.6.0` remains the public release users get by default. If `main` temporarily contains a final public version during the release flow, the internal preview workflow skips publishing and lets the tag-driven public workflow handle that commit. ## Publishing Flow Internal preview: 1. Merge a PR to `main`. 2. `.github/workflows/test.yml` runs. 3. `.github/workflows/integration-eval.yml` runs a real rollout after the tested `main` commit passes. 4. `.github/workflows/internal-preview-release.yml` publishes to PyPI only if the integration gate passed. The integration gate selects the first configured live LLM provider that can answer a small probe request, then uses that same provider for the smoke rollout and agent judge. Configure at least one of `DEEPSEEK_API_KEY`, `GLM_API_KEY`, `QWEN_API_KEY`, `LITELLM_API_KEY`/`BF_TOKEN`, `OPENAI_API_KEY`, or `GITHUB_MODELS_TOKEN` as GitHub Actions secrets for the job environment. `DAYTONA_API_KEY` is optional and enables the Daytona parity and reaper checks. Public release: 1. Update `pyproject.toml` from the next `.dev0` version to the final public version, for example `0.6.1.dev0 -> 0.6.1`. 2. Merge the release PR to `main`. 3. Push a matching tag, for example `v0.6.1`. 4. `.github/workflows/public-release.yml` validates the tag, publishes to PyPI, and creates a GitHub Release. The workflow refuses tags whose commits are not contained in `origin/main`. 5. Bump `main` to the next `.dev0`, for example `0.6.2.dev0`. ## One-Time PyPI Setup Configure PyPI Trusted Publishing for the `benchflow` project. No PyPI token is stored in GitHub. Create these PyPI trusted publishers: | Channel | Repository | Workflow filename | Environment | | ---------------- | ------------------------ | ------------------------------ | ----------------------- | | Internal preview | `benchflow-ai/benchflow` | `internal-preview-release.yml` | `pypi-internal-preview` | | Public | `benchflow-ai/benchflow` | `public-release.yml` | `pypi-public` | Create matching GitHub environments: * `pypi-internal-preview`: used for automatic preview publishing from `main`. * `pypi-public`: used for tag-driven public releases. The workflows build with `uv build --no-sources`, check distributions with `twine check`, and publish with `uv publish`. # 2026 06 09 task standard validation Source: https://docs.benchflow.ai/reports/2026-06-09-task-standard-validation # Task standard — validation evidence Date: 2026-06-09 This page records how the BenchFlow task standard (`docs/task-standard.md`) was validated: that a single `task.md` package can **represent**, **run**, and **score** real benchmarks end-to-end. SkillsBench is the worked example because it exercises the full surface — environment build, skill injection, oracle, and verifier. The bar throughout is **"runs end-to-end with a real rollout and a produced verdict,"** not "the agent passes." Pass rate is a property of the model under test, not of the standard. ## 1. Representation — conversion parity 88 / 88 (deterministic) For all 88 SkillsBench `main` tasks, a roundtrip conformance check migrates each legacy split package → native `task.md` → exports back to split, and compares the canonical task config, the normalized prompt, and SHA-256 file maps for the `environment/`, `solution/`, and `tests/` trees. ``` === SkillsBench conversion-parity: 88 tasks | PASS=88 MISMATCH=0 ERROR=0 === ``` Every task is representable as native `task.md` with **zero** semantic loss on the compatibility surface. ## 2. Scoring — oracle E2E parity 6 / 6 (Docker) Each sampled task runs twice with the deterministic oracle agent (`--agent oracle`, no LLM — it executes the reference solution), once on the legacy layout and once on the pure-native package (`task.md` + `oracle/` + `verifier/`). Reward must match. | Task | legacy | task.md | parity | | ---------------------------------- | -----: | ------: | ---------------------------- | | tictoc-unnecessary-abort-detection | 1.0 | 1.0 | ✅ | | llm-prefix-cache-replay | 1.0 | 1.0 | ✅ | | parallel-tfidf-search | 1.0 | 1.0 | ✅ | | grid-dispatch-operator | 1.0 | 1.0 | ✅ | | travel-planning | 0.0 | 0.0 | ✅ (oracle fails identically) | | 3d-scan-calc | 0.0 | 0.0 | ✅ (oracle fails identically) | The native package builds the environment, runs the oracle, and runs the verifier end-to-end, scoring **identically** to the legacy layout. The two `0.0` rows are the upstream oracle itself failing under the hardened sandbox — independent of the standard; parity is preserved. ## 3. Live agent run — three interaction modes To show the packages execute under a real agent (not just the deterministic oracle), five converted tasks were run with a live agent across all three SkillsBench interaction modes — **no-skill**, **with-skill**, and **self-gen** (the agent authors a skill, then solves with it). A rollout is counted only when it produced real work (tokens spent, tool calls made, and a verdict returned). **12 of 15 mode-cells ran clean** (4 of 5 tasks across 3 skill modes; the fifth fails for non-task.md infrastructure reasons), each a real rollout. The skill machinery genuinely engages: skills are injected and invoked, not merely declared. All clean rewards were `0.0` — expected for the small model used here; these tasks sit below its capability floor, so the result to read is *the packages run and the skill path fires in every mode*, not the absolute score. The failing task's three mode-cells failed for environment/verifier-portability reasons unrelated to the conversion. ## What this shows / does not show * **Shows:** `task.md` losslessly represents 100% of SkillsBench, scores with exact oracle parity to the legacy layout, and runs end-to-end under a live agent in all three interaction modes on 4 of the 5 sampled tasks (the fifth fails for non-task.md infrastructure reasons). * **Does not show:** a measurable skill *uplift* — that needs a base model strong enough to sometimes pass; the runs here establish the plumbing, not the lift. # Running benchmarks Source: https://docs.benchflow.ai/running-benchmarks # Running Adapted Benchmarks How to run benchmarks that have been converted into Harbor-format tasks for BenchFlow. BenchFlow ships with adapted benchmarks under `benchmarks//`. Each benchmark includes a converter, parity tests, metadata, and one or more YAML job configs. This guide covers how to run them — from a single task to a full evaluation sweep. > \[!NOTE] > BenchFlow is providing first-party support for PrimeIntellect Verifiers and OpenReward Standard. > **Working inside the benchflow repo?** Use `uv run bench` instead of `bench` > to run the CLI from your local editable install. *** ## Available benchmarks | Benchmark | Tasks | Verification | Config | | ---------------------------------------------------------- | ----- | ---------------------------- | ------------------------------------------------------------ | | [Harvey LAB](https://github.com/harveyai/harvey-labs) | 1,251 | LLM-as-judge (per-criterion) | `benchmarks/harvey-lab/` | | [ProgramBench](https://programbench.com) | 201 | Deterministic unit tests | `benchmarks/programbench/` | | [SkillsBench](https://github.com/benchflow-ai/skillsbench) | 94+ | Unit tests | `--source-repo benchflow-ai/skillsbench --source-path tasks` | Each adapted benchmark includes: * **`benchflow.py`** — converter for the raw benchmark source * **`benchmark.yaml`** — metadata descriptor (task count, categories, verification method, parity results) * **`-*.yaml`** — job configs for different agents/models * **`parity_test.py`** — parity validation suite * **`parity_experiment.json`** — recorded parity results ### Environment-plane benchmarks Stateful, multi-service benchmarks integrate differently: instead of a converter they ship an `environment.toml` **manifest** and run on the [Environment plane](./environment-plane.md). Two are onboarded: | Benchmark | Topology | Manifest | | -------------- | ------------------------------------------------------- | ---------------------------------------- | | **ClawsBench** | mock Gmail/Slack/Calendar/Docs/Drive, framework-started | `benchmarks/clawsbench/environment.toml` | | **chi-bench** | \~25k-LOC healthcare simulator, image-owned lifecycle | `benchmarks/chi-bench/environment.toml` | See [Running a benchmark with an Environment manifest](#running-a-benchmark-with-an-environment-manifest) below. *** ## Quick start ### Option 1: YAML config (`bench eval create --config`) The simplest path. Point at a YAML config that specifies the benchmark source, agent, and model: ```bash theme={null} GEMINI_API_KEY=... bench eval create --config benchmarks/harvey-lab/harvey-lab-gemini-flash-lite.yaml GEMINI_API_KEY=... bench eval create --config benchmarks/programbench/programbench-gemini-flash-lite.yaml bench eval create --config benchmarks/harvey-lab/harvey-lab-gemini-flash-lite.yaml ``` The config handles everything — downloads/generates tasks, resolves the task path, and runs the evaluation. ### Option 2: CLI flags Use CLI flags for ad-hoc runs without a config file: ```bash theme={null} # Harvey LAB — single pre-converted task bench eval create \ --source-repo benchflow-ai/benchmarks \ --source-path datasets/harvey-lab/tasks/corporate-ma-analyze-cim-deal-teaser-scenario-01 \ --agent gemini --model gemini-3.1-flash-lite-preview --sandbox docker # Harvey LAB harness adapter smoke test. # Requires GEMINI_API_KEY for the agent and ANTHROPIC_API_KEY for the verifier. uv run bench eval create \ --source-repo benchflow-ai/benchmarks \ --source-path datasets/harvey-lab/tasks/corporate-ma-analyze-cim-deal-teaser-scenario-01 \ --agent harvey-lab-harness \ --model gemini-3.1-flash-lite-preview \ --sandbox docker \ --concurrency 1 \ --jobs-dir jobs/smoke-test/harvey-harness # Harvey LAB — all pre-converted tasks bench eval create \ --source-repo benchflow-ai/benchmarks \ --source-path datasets/harvey-lab/tasks \ --agent gemini --model gemini-3.1-flash-lite-preview --sandbox docker --concurrency 4 # SkillsBench bench eval create \ --source-repo benchflow-ai/skillsbench \ --source-path tasks/edit-pdf \ --agent gemini --model gemini-3.1-flash-lite-preview # ProgramBench — single task (tasks are generated at runtime by the converter; # see "Running ProgramBench" below for the generation step) bench eval create \ --tasks-dir benchmarks/programbench/tasks/abishekvashok__cmatrix.5c082c6 \ --agent gemini --model gemini-3.1-flash-lite-preview --sandbox docker # Claude Code on Daytona bench eval create \ --source-repo benchflow-ai/skillsbench \ --source-path tasks \ --agent claude-agent-acp --model anthropic/claude-sonnet-4-6 --sandbox daytona --concurrency 32 ``` > **Note:** Harvey LAB task names in `benchflow-ai/benchmarks` are flattened with > hyphens (e.g. `corporate-ma-analyze-cim-deal-teaser-scenario-01`), not nested > paths like the original repo (`corporate-ma/analyze-cim-deal-teaser/scenario-01`). ### Option 3: Python API For programmatic use, custom pipelines, or integration with other tools: ```python theme={null} import asyncio from benchflow.evaluation import Evaluation async def main(): job = Evaluation.from_yaml("benchmarks/harvey-lab/harvey-lab-gemini-flash-lite.yaml") result = await job.run() print(f"Score: {result.passed}/{result.total} ({result.score:.1%})") asyncio.run(main()) ``` For single-task runs: ```python theme={null} import benchflow as bf from benchflow import RolloutConfig, Scene from benchflow._utils.benchmark_repos import resolve_source task_path = resolve_source("benchflow-ai/skillsbench", path="tasks/edit-pdf") config = RolloutConfig( task_path=task_path, scenes=[Scene.single(agent="gemini", model="gemini-3.1-flash-lite-preview")], environment="docker", ) result = await bf.run(config) print(result.rewards) ``` *** ## Versioned dataset runs (`--dataset`) For runs whose results should be attributable to a published, immutable dataset version (leaderboards, papers, release evidence), resolve the task set from a dataset registry instead of pointing at a directory or branch: ```bash theme={null} # Resolve skillsbench@1.1 from the registry, verify every task's content # digest against the pinned snapshot, then run. bench eval create -d skillsbench@1.1 \ --agent claude-agent-acp --model claude-haiku-4-5-20251001 # Versions are immutable, so the version is always explicit — there is no # floating "latest". --include/--exclude filter the registry roster. bench eval create -d skillsbench@1.1 --include xlsx-recover-data ... ``` A registry (`registry.json` at a dataset repo's root — see skillsbench's [`docs/dataset-versioning.md`](https://github.com/benchflow-ai/skillsbench/blob/main/docs/dataset-versioning.md)) pins each dataset version to an exact `git_commit_id` and per-task sha256 content digests. Resolution clones the pinned commit into `.cache/datasets`, materializes an immutable per-commit snapshot, recomputes every task's digest, and **fails before running anything** on any mismatch. Snapshot directories that are not part of the registry entry are excluded from the run. The entry's `bench_version` range is a hard gate: running on a benchflow outside the range the dataset was validated against fails before anything runs, because the results may not be comparable with published runs. `--ignore-bench-version` overrides the gate for local experimentation — the run then proceeds with a visible warning on the record. The registry is fetched from the skillsbench repo by default; point `--registry` at another URL or a local `registry.json` to override. Every `result.json`/`config.json` is stamped with `dataset_name`, `dataset_version`, and the per-task `task_digest` (`summary.json` carries the name/version), so downstream tooling can group results by `dataset@version`. `--tasks-dir` stays as the visibly distinct dev mode: its artifacts carry no dataset fields — but they still stamp a live-computed `task_digest`, so even dev trajectories remain attributable to exact task content. `bench tasks digest ` prints the same digest for any task directory. *** ## Running a subset of tasks ### Single task ```bash theme={null} bench eval create \ --source-repo benchflow-ai/skillsbench \ --source-path tasks/edit-pdf \ --agent gemini --model gemini-3.1-flash-lite-preview bench eval create \ --tasks-dir benchmarks/programbench/tasks/abishekvashok__cmatrix.5c082c6 \ --agent gemini --model gemini-3.1-flash-lite-preview --sandbox docker bench eval create \ --tasks-dir .cache/harvey-lab-tasks/corporate-ma-review-data-room-red-flag-review \ --agent gemini --model gemini-3.1-flash-lite-preview --sandbox docker ``` ### Batch with a tasks directory Point `bench eval create --tasks-dir` at a directory containing only the tasks you want: ```bash theme={null} bench eval create --tasks-dir benchmarks/programbench/tasks \ --agent gemini --model gemini-3.1-flash-lite-preview --sandbox docker --concurrency 4 ``` ### Using `--source-path` for remote subsets ```bash theme={null} # SkillsBench single task bench eval create \ --source-repo benchflow-ai/skillsbench \ --source-path tasks/edit-pdf \ --agent gemini --model gemini-3.1-flash-lite-preview --sandbox docker # Harvey LAB single task (pre-converted) bench eval create \ --source-repo benchflow-ai/benchmarks \ --source-path datasets/harvey-lab/tasks/corporate-ma-analyze-cim-deal-teaser-scenario-01 \ --agent gemini --model gemini-3.1-flash-lite-preview --sandbox docker ``` *** ## Running ProgramBench 201 program-reconstruction tasks across 7 languages (C, Rust, Go, C++, Java, Haskell, Bash). Tasks are **generated** at runtime from the ProgramBench repo's metadata — `benchmarks/programbench/tasks/` is not checked into this repo and must be produced first. ### Prerequisites * Docker (images are linux/amd64 only — use a Linux x86\_64 machine) * \~20GB disk for Docker images * Internet access for HuggingFace test blob downloads during verification * A local clone of [`programbench`](https://programbench.com) (passed via `--programbench-dir` to the generator) ### Generate the tasks ```bash theme={null} # All 200 tasks python -m benchmarks.programbench.main \ --programbench-dir ~/programbench \ --output-dir benchmarks/programbench/tasks # Or a single task python -m benchmarks.programbench.main \ --programbench-dir ~/programbench \ --output-dir benchmarks/programbench/tasks \ --task-ids abishekvashok__cmatrix.5c082c6 ``` ### Run all tasks ```bash theme={null} bench eval create --config benchmarks/programbench/programbench-gemini-flash-lite.yaml ``` ### Run a single task (after generation) ```bash theme={null} bench eval create \ --tasks-dir benchmarks/programbench/tasks/abishekvashok__cmatrix.5c082c6 \ --agent gemini --model gemini-3.1-flash-lite-preview --sandbox docker ``` ### Oracle verification Verify a task is solvable using the gold solution (original source at commit): ```bash theme={null} bench eval create \ --tasks-dir benchmarks/programbench/tasks/abishekvashok__cmatrix.5c082c6 \ --agent oracle --sandbox docker ``` ### Validate a task directory ```bash theme={null} bench tasks check benchmarks/programbench/tasks/abishekvashok__cmatrix.5c082c6 ``` *** ## Choosing an agent Any registered BenchFlow agent works with adapted benchmarks. List them: ```bash theme={null} bench agent list ``` Common choices: | Agent | Key | Auth | | ------------------ | ------------------------------------------ | ---------------------------------------------------------------------- | | Gemini | `gemini` | `GEMINI_API_KEY` or host login | | Claude Code | `claude-agent-acp` (alias: `claude`) | `ANTHROPIC_API_KEY` or host login | | Codex | `codex-acp` (alias: `codex`) | `OPENAI_API_KEY`, `CODEX_API_KEY`, `CODEX_ACCESS_TOKEN`, or host login | | OpenHands | `openhands` (alias: `oh`) | `LLM_API_KEY` | | Harvey LAB harness | `harvey-lab-harness` (alias: `harvey-lab`) | Provider key matching model | The auth column shows each agent's native/default credentials. Provider-prefixed models can use provider-specific credentials instead; for example, Azure Foundry models use `AZURE_API_KEY` plus `AZURE_API_ENDPOINT` with prefixes such as `azure-foundry-openai/gpt-5.5` or `azure-foundry-anthropic/claude-opus-4-5`. Any agent can also be run via [ACPX](https://acpx.sh/) by prefixing with `acpx/`: ```bash theme={null} bench eval create --tasks-dir tasks/edit-pdf --agent acpx/gemini --model gemini-3.1-flash-lite-preview --sandbox daytona ``` ACPX is a headless ACP client that adds persistent sessions and crash recovery. The underlying agent's install, env vars, credentials, and skill paths are all preserved. The **Harvey LAB harness** agent is special — it runs Harvey LAB's own agent loop (6 tools, system prompt) inside BenchFlow's sandbox. Use it for parity testing (same agent on both original and converted tasks). *** ## Choosing a sandbox | Sandbox | Flag | Best for | | ------- | ------------------- | ----------------------------------------------------- | | Docker | `--sandbox docker` | Local development, small runs (≤10 tasks) | | Daytona | `--sandbox daytona` | Cloud runs with concurrency (needs `DAYTONA_API_KEY`) | | Modal | `--sandbox modal` | Serverless, high concurrency (needs Modal auth) | For large-scale runs (100+ tasks), use Daytona or Modal with high concurrency: ```bash theme={null} bench eval create \ --source-repo benchflow-ai/skillsbench \ --source-path tasks \ --agent gemini --model gemini-3.1-flash-lite-preview --sandbox daytona --concurrency 64 ``` > **Daytona has a 10 GB-per-sandbox hard cap.** Tasks with heavy images (large > HuggingFace model snapshots, Playwright, LaTeX/marker — e.g. > `latex-formula-extraction`) overflow during bootstrap (`No space left on > device`) or hang at "Sandbox user agent ready" with no trajectory. Run those on > `--sandbox docker` (host disk, no cap); keep Daytona for lighter tasks. *** ## SkillsBench skill-toggle matrix (Opus-4.8 + Gemini) on Daytona A self-contained recipe for the four-cell matrix of **`{Opus-4.8 via Bedrock, Gemini-3.5-flash}` × `{with-skills, without-skills}`**, agent `openhands`, sandbox `daytona`. Each cell produces a complete trajectory (`trajectory/{acp,llm}_trajectory.jsonl`) plus a verifier reward — but treat a cell as done only after the audit in [Verifying the batch](#verifying-the-batch) passes. ### Setup (once per shell) ```bash theme={null} # 1. Run the CLI from a benchflow checkout. BenchFlow starts LiteLLM as the # provider gateway for Bedrock/Gemini/Azure/etc. Inside the repo, use `uv run bench`. cd /path/to/benchflow # 2. A local skillsbench clone, so --skills-dir can point at a task's bundled skills git clone https://github.com/benchflow-ai/skillsbench export SKILLSBENCH=$PWD/skillsbench # 3. Credentials — verify each LIVE first. A dead key shows up only as # `openhands ACP error -32603` on the first model call, never a clean auth error: # Bedrock: a `converse` call with `Authorization: Bearer $AWS_BEARER_TOKEN_BEDROCK` -> 200 # Gemini: `.../v1beta/models/:generateContent?key=$GEMINI_API_KEY` -> 200 export AWS_BEARER_TOKEN_BEDROCK=... export AWS_REGION=us-west-2 AWS_DEFAULT_REGION=us-west-2 export GEMINI_API_KEY=... # 4. MAX thinking for Opus-4.8 (opt-in). WITHOUT this the run uses the agent's # default effort = adaptive-thinking `high`, NOT max. LiteLLM receives this # env var on both Daytona and Docker. export BENCHFLOW_BEDROCK_THINKING_EFFORT=max # 5. Strip stale external gateway vars; BenchFlow will generate its own LiteLLM config: unset LLM_BASE_URL LLM_API_KEY OPENAI_BASE_URL BENCHFLOW_PROVIDER_BASE_URL \ BENCHFLOW_PROVIDER_API_KEY LITELLM_BASE_URL LITELLM_API_KEY ``` > Pick a **light** task — Daytona caps each sandbox at 10 GB (see the note above). > `citation-check` is a good default; heavy tasks need `--sandbox docker`. Note that > MAX effort makes each Opus turn much slower (deep server-side reasoning — a > `citation-check` cell took \~10–15 min at `max` vs \~3 min at the default effort). ### Run the four cells ```bash theme={null} TASK=citation-check COMMON="--tasks-dir $SKILLSBENCH/tasks --include $TASK --agent openhands \ --sandbox daytona --concurrency 1 --usage-tracking required --agent-idle-timeout none" # (1) Opus-4.8 (MAX) — with skills bench eval create $COMMON --model aws-bedrock/us.anthropic.claude-opus-4-8 \ --skill-mode with-skill --jobs-dir jobs/opus-skill # (2) Opus-4.8 (MAX) — without skills bench eval create $COMMON --model aws-bedrock/us.anthropic.claude-opus-4-8 \ --skill-mode no-skill --jobs-dir jobs/opus-noskill # (3) Gemini-3.5-flash — with skills bench eval create $COMMON --model gemini-3.5-flash --agent-env LLM_CACHING_PROMPT=false \ --skill-mode with-skill --jobs-dir jobs/gemini-skill # (4) Gemini-3.5-flash — without skills bench eval create $COMMON --model gemini-3.5-flash --agent-env LLM_CACHING_PROMPT=false \ --skill-mode no-skill --jobs-dir jobs/gemini-noskill ``` `BENCHFLOW_BEDROCK_THINKING_EFFORT=max` is what makes the two Opus cells actually run at MAX. LiteLLM writes the provider call metadata to `trajectory/llm_trajectory.jsonl`; confirm the adaptive thinking effort there. | Model (`--model`) | Skills | Cell-specific flags | | ------------------------------------------ | ------- | -------------------------------------------------------------- | | `aws-bedrock/us.anthropic.claude-opus-4-8` | with | `--skill-mode with-skill` | | `aws-bedrock/us.anthropic.claude-opus-4-8` | without | `--skill-mode no-skill` | | `gemini-3.5-flash` | with | `--agent-env LLM_CACHING_PROMPT=false --skill-mode with-skill` | | `gemini-3.5-flash` | without | `--agent-env LLM_CACHING_PROMPT=false --skill-mode no-skill` | ### Verifying the batch A finished command is **not** a healthy trial. After each batch, audit the trajectories with the **`benchflow-experiment-review`** skill (repo copy at `.agents/skills/benchflow-experiment-review`, also reachable through the `.claude/skills` symlink; see the Experiment-guidance notes in `AGENTS.md`). A trial counts as healthy only when **every** check passes: complete trajectory + metadata (timing, token usage, tool usage), correct pass/fail/timeout status, verifier isolation (verifier starts after the agent exits), no reward hacking, and the right skill posture — with-skills cells must show the task skill loaded (`task_skills_loading: 1`), without-skills cells must not (`task_skills_loading: 0`, and the task skill absent from the trajectory; generic openhands built-ins such as `.agents/skills` / `invoke_skill` appear in *every* run and are **not** leakage). Quick smoke checks before the full audit (per jobs-dir): ```bash theme={null} J=jobs/opus-skill find $J -name rewards.jsonl -exec tail -1 {} \; # reward grep -ho '"usage_source": "[a-z_]*"' $(find $J -name result.json) # expect provider_response grep -ho '"effort": "[a-z]*"' $(find $J -name llm_trajectory.jsonl) | sort -u # Opus MAX -> "max" python3 .agents/skills/benchflow-experiment-review/scripts/extract_harness_skills.py \ "$(find $J -name llm_trajectory.jsonl | head -1)" --task-skill ``` Notes: * `--usage-tracking required` records provider-reported token usage into each trajectory. * `--agent-idle-timeout none` disables the idle watchdog (the task wall-clock still applies). * Opus-4.8 on Bedrock needs the adaptive-thinking patch, which LiteLLM loads into its proxy process (`src/benchflow/providers/litellm_bedrock_patch.py`); see `AGENTS.md`. * For heavy tasks, replace `--sandbox daytona` with `--sandbox docker` — same flags otherwise. *** ## Running a benchmark with an Environment manifest A **stateful** benchmark — one with mock services, databases, or accounts the agent acts on — declares its world in an `environment.toml` manifest and runs on the [Environment plane](./environment-plane.md). Use `bench eval create --tasks-dir ...` for both single-task and batch manifest-backed evaluations; `--environment-manifest` applies the manifest to every rollout in the Job pipeline. ```bash theme={null} # single task bench eval create --tasks-dir benchmarks/clawsbench/tasks/ \ --environment-manifest benchmarks/clawsbench/environment.toml \ --agent claude-agent-acp --model claude-haiku-4-5 bench eval create --tasks-dir benchmarks/chi-bench/tasks/ \ --environment-manifest benchmarks/chi-bench/environment.toml \ --agent claude-agent-acp --model claude-haiku-4-5 # batch via the Job API bench eval create --tasks-dir benchmarks/clawsbench/tasks \ --environment-manifest benchmarks/clawsbench/environment.toml \ --agent claude-agent-acp --model claude-haiku-4-5 ``` YAML configs may declare the same seam with `environment_manifest: ` at the top level so the batch run is reproducible from disk. `--environment-manifest` is distinct from `--sandbox`: the sandbox is *where* the rollout runs; the environment manifest is *the world* the agent acts in. BenchFlow provisions the environment, gates on its readiness before the agent runs, and tears it down afterward. See [the Environment plane](./environment-plane.md) for the full manifest schema, both onboarded benchmarks, and the `snapshot`/`restore` roll-back contract. *** ## Running foreign benchmarks (inbound adapters) BenchFlow runs benchmarks authored in other formats without converting them first. An **inbound adapter** translates a foreign task directory into BenchFlow-native shape; the rollout then runs natively: | Source format | Signature file | Adapter | | ------------- | -------------- | --------------- | | Harbor | `task.toml` | `HarborAdapter` | `benchflow.adapters.inbound.detect_adapter()` sniffs a task directory and picks the adapter whose format it matches. The adapter is a pure `Path -> InboundTask` translation: it reads a directory and returns an in-memory native task, building no sandboxes and running nothing. *** ## Continual learning (`sequential-shared` job mode) By default a job runs its rollouts concurrently and isolated (`parallel-independent`). A **continual-learning** job instead runs them strictly in order over one persistent, versioned store of memory + skills — set `job_mode: sequential-shared` in the YAML config: ```yaml theme={null} source: repo: benchflow-ai/skillsbench path: tasks agent: claude-agent-acp model: claude-haiku-4-5 job_mode: sequential-shared ``` In this mode each rollout reads the current `LearnerStore` state and, after it scores, offers its reward as a learning-curve metric: an improvement stamps a new generation, a regression is reverted to the best generation so far. Concurrency is ignored — a shared mutable store cannot be written by overlapping rollouts. See the [architecture doc](./architecture.md#the-eight-capabilities--how-each-fits), capability 5, for the full design. *** ## Reading results Results land under `jobs///`: ``` jobs/ └── harvey-lab-gemini-2026-05-06/ ├── corporate-ma-review-data-room-red-flag-review/ │ ├── result.json # verifier output (reward, passed criteria) │ └── trajectory/ │ └── acp_trajectory.jsonl # full agent trace ├── real-estate-extract-psa-key-terms-scenario-01/ │ ├── result.json │ └── trajectory/ └── ... ``` The `result.json` contains (abridged): ```json theme={null} { "rewards": {"reward": 0.48}, "n_tool_calls": 12, "n_skill_invocations": 2, "agent_result": {"total_tokens": 23993, "cost_usd": 0.07, "usage_source": "provider_response"}, "final_metrics": {"total_prompt_tokens": 18000, "total_completion_tokens": 5993}, "error": null, "verifier_error": null } ``` **Canonical fields — read these, not invented top-level ones.** The reward, token totals, and outcome each live in exactly one place: | You want | Read | Notes | | ---------------- | -------------------------------------------------------- | ------------------------------------------------------------- | | reward | `rewards.reward` | scalar 0.0–1.0, or `null` if unscored | | token total | `agent_result.total_tokens` | `null` when no provider usage was captured (e.g. hosted runs) | | outcome / status | derived from `rewards.reward` + `error`/`verifier_error` | not stored as a field; see below | There is intentionally **no top-level `reward`, `total_tokens`, or `status` key** — those names are absent, not `null`. A naive consumer doing `result["reward"]` or `result.get("total_tokens")` gets `None` because the key does not exist, never because the value is null. Pass/fail is a *derived* classification (only `reward == 1.0` passes); BenchFlow computes it from `rewards.reward` plus the error channels rather than persisting a redundant `status`. The same nested shape is produced for both native rollouts and hosted-env runs, so one reader handles both. `n_skill_invocations` is derived from structured ACP trajectory events: BenchFlow counts only `tool_call` events whose `kind` is `skill`. Job `summary.json` also includes `total_skill_invocations` and `avg_skill_invocations` across the rollouts in the run. List evaluations: ```bash theme={null} bench eval list jobs/ ``` *** ## Running parity validation Parity validation is a **developer/maintainer workflow** for verifying that an adapter preserves benchmark semantics. These scripts live under each benchmark's directory: ```bash theme={null} uv run python benchmarks/harvey-lab/parity_test.py \ --mode full \ --harvey-root .cache/datasets/harveyai/harvey-labs ANTHROPIC_API_KEY=... uv run python benchmarks/harvey-lab/parity_test.py \ --mode eval-parity GEMINI_API_KEY=... uv run python benchmarks/harvey-lab/parity_test.py \ --mode side-by-side ``` Recorded parity results are in `parity_experiment.json` and `benchmark.yaml`. *** ## YAML config reference Job configs use the two-field `source` pattern to reference remote benchmark repos: ```yaml theme={null} # Example: SkillsBench config — direct from remote repo source: repo: benchflow-ai/skillsbench # GitHub repo (org/repo) path: tasks # subpath within the repo ref: main # branch/tag (optional) agent: claude-agent-acp # agent from registry model: zai/glm-5.1 # model ID environment: daytona # sandbox concurrency: 8 # parallel tasks ``` All adapted benchmarks use the same `source` pattern, pointing at the [benchmarks dataset repo](https://github.com/benchflow-ai/benchmarks): ```yaml theme={null} # benchmarks/harvey-lab/harvey-lab-gemini-flash-lite.yaml source: repo: benchflow-ai/benchmarks path: datasets/harvey-lab/tasks agent: gemini model: gemini/gemini-3.1-flash-lite-preview environment: docker concurrency: 4 ``` ```yaml theme={null} # benchmarks/programbench/programbench-gemini-flash-lite.yaml source: repo: benchflow-ai/benchmarks path: datasets/programbench/tasks agent: gemini model: gemini-3.1-flash-lite-preview environment: docker concurrency: 4 ``` You can also use `tasks_dir:` for local paths: ```yaml theme={null} tasks_dir: ./my-local-tasks agent: gemini model: gemini/gemini-3.1-flash-lite-preview ``` All fields from [CLI reference](./reference/cli.md#yaml-config-format) apply: `source`, `tasks_dir`, `agent`, `model`, `environment`, `concurrency`, `sandbox_setup_timeout`, `skills_dir`, `agent_env`, `max_retries`. *** ## Adding a new benchmark See the [Benchmark Conversion Guide](../benchmarks/CONVERT.md) for the 9-step process to convert a new benchmark into Harbor-format tasks for BenchFlow. Harvey LAB (`benchmarks/harvey-lab/`) and ProgramBench (`benchmarks/programbench/`) are reference implementations. # Sandbox hardening Source: https://docs.benchflow.ai/sandbox-hardening # Sandbox hardening ## Why Frontier agents will reward-hack any verifier they can. [BenchJack](https://rdi.berkeley.edu/blog/trustworthy-benchmarks-cont/) (Berkeley RDI, April 2026) showed that \~100% of tasks across eight major AI agent benchmarks are solvable by exploiting the *verifier's runtime* rather than the agent's intelligence: pytest `conftest.py` hooks, planted PATH binaries, leaked answer keys, `.pth` injection, `sitecustomize.py` shadowing. [Meerkat](https://debugml.github.io/cheating-agents/) (Penn, same week) confirmed it from the trace side — frontier models discover and use these shortcuts in the wild. benchflow's verifier hardening blocks these patterns by default. Tasks opt out individually when their legitimate test setup requires a hook the cleanup would otherwise delete (e.g. qutebrowser's `conftest.py` resolves a real circular import). ## What gets hardened Between the agent's last action and the verifier's first command, `harden_before_verify()` (in `src/benchflow/sandbox/lockdown.py`) runs this sequence: 1. **Kill sandbox-user processes** — `pkill -u `, twice with a sleep, so the agent can't keep writing during teardown. 2. **Wipe `/logs/verifier`** — fresh, root-owned, mode 0777 (so a non-root verifier can write its own results). 3. **Restore build-config files** from a pre-agent snapshot — `setup.py`, `pyproject.toml`, `setup.cfg`, `tox.ini`, `noxfile.py`, `hatch.toml`, `flit.ini`, `MANIFEST.in`, `requirements.txt`, `requirements-dev.txt`, `Makefile`. Closes the editable-install hijack vector. 4. **Optionally restore the entire workspace** from `/testbed_verify` snapshot — opt-in per task because some tasks score workspace edits. 5. **Purge symlinks whose realpath escapes the workspace** — agent can't redirect to `/tmp/evil.py`. 6. **Purge `__pycache__` trees** that didn't exist in the pre-agent baseline. 7. **chown workspace to root** — belt-and-suspenders against zombie sandbox-user processes. 8. **`CLEANUP_CMD`** — delete `conftest.py` outside `/tests/`, `*.py` from `/tmp`/`/var/tmp`, `sitecustomize.py`/`usercustomize.py`/`*.pth` from writable `sys.path` entries. 9. **Set `VERIFIER_ENV`** — pinned `PATH`, `PYTEST_ADDOPTS=-c /dev/null --confcutdir=/tests --rootdir=/app -p no:cacheprovider`, `PYTHONPATH=""`, `PYTHONDONTWRITEBYTECODE=1`, plugin auto-discovery off. 10. **Discover allowed pytest plugins** — `importlib.metadata` scan inside the container, then merge with task-declared `pytest_plugins` from `task.toml`. Anything not in the allow-list is blocked. The verifier then runs against this hardened workspace. ## Per-task opt-outs Tasks declare opt-outs in `task.toml`: ```toml theme={null} [verifier.hardening] cleanup_conftests = false ``` | Flag | Default | Effect when `false` | | ------------------- | ------- | ---------------------------------------------------------- | | `cleanup_conftests` | `true` | Don't delete `conftest.py` outside `/tests/` before verify | Other cleanup steps (`sitecustomize.py`, `.pth`, `/tmp` `*.py`) always run — they have no legitimate use in a test artifact and disabling them would broaden the attack surface beyond what real tasks need. Unknown keys in `[verifier.hardening]` are warned and ignored. String values for boolean flags are rejected. See [`progressive-disclosure.md`](./progressive-disclosure.md#per-task-hardening-opt-outs) for the qutebrowser case study (legitimate `conftest.py` for circular-import fix). ## Validation: the labs Two runnable research artifacts demonstrate the hardening's effectiveness. They are historical (0.2.x-era) and archived under [`docs/labs/`](https://github.com/benchflow-ai/benchflow/tree/main/docs/labs); the defenses they validate remain in place. ### `docs/labs/benchjack-sandbox-hardening/` End-to-end demo that benchflow 0.2.1+ blocks three BenchJack-style exploits that work against 0.2.0. Each exploit flips the reward from 0.0 to 1.0 under 0.2.0 — under hardened benchflow, the same exploits fail and reward stays 0.0. Patterns covered: * `pattern1_conftest_hook/` — agent plants `conftest.py` with a pytest\_runtest\_makereport hook that overwrites the report * `pattern2_answer_lookup/` — agent reads `/solution/` directly * `pattern7_pth_injection/` — agent installs a `.pth` file that imports a sitecustomize-style shadow One-command repro: `cd docs/labs/benchjack-sandbox-hardening && ./run_comparison.py`. Output is `comparison.ipynb`-style diff: 0.2.0 reward=1.0 / hardened reward=0.0 per pattern. ### `docs/labs/reward-hack-matrix/` Full reward-hack sweep across real benchmark tasks comparing 0.2.0 vs 0.2.2. Scope: 8 major benchmarks, \~100 tasks, multiple exploit families. Output: a sweep JSON (`sweep_0.2.0_vs_0.2.2.json`) showing the hardening rate by exploit class. Run with `cd docs/labs/reward-hack-matrix && python run_matrix.py`. The lab README documents the methodology and per-task results. ## Threat model and known gaps Benchflow's hardening assumes: * The Docker image and pre-installed packages are trusted (they're built before the agent runs). * The agent runs as a non-root sandbox user (`sandbox_user`) — when set to `None`, hardening still runs but the agent can defeat lockdown by being root. Don't run untrusted agents as root. * The verifier itself is trusted code shipped with the task; benchflow only hardens *what the agent can do to the verifier's environment*. Known residual risk: * An agent with sustained access can poison `__pycache__` for files that exist in the baseline (those caches aren't deleted because some tasks diff workspace against `/testbed_verify`). Mitigated by the workspace chown but not eliminated. * Tasks that don't ship a build-config snapshot can still be hijacked via `setup.py` edits. Snapshot is automatic for declared filenames — task authors don't need to opt in. ## Related * [`docs/labs/benchjack-sandbox-hardening/README.md`](https://github.com/benchflow-ai/benchflow/tree/main/docs/labs/benchjack-sandbox-hardening) — full BenchJack pattern catalog and repro instructions (historical, 0.2.x-era). * [`docs/labs/reward-hack-matrix/README.md`](https://github.com/benchflow-ai/benchflow/tree/main/docs/labs/reward-hack-matrix) — methodology, exploit taxonomy, sweep results (historical, 0.2.x-era). * [`progressive-disclosure.md`](./progressive-disclosure.md) — soft-verify (the relaxed hardening used between rounds in multi-round trials). * [`task-authoring.md`](./task-authoring.md) — the `task.toml` schema including `[verifier.hardening]` opt-outs. # Skill eval Source: https://docs.benchflow.ai/skill-eval # Skill evals Test whether your agent skill actually helps agents perform better. ## Install ```bash theme={null} uv tool install --prerelease allow benchflow # --prerelease allow is for the pinned LiteLLM rc dependency, not benchflow itself. ``` ## Overview `bench skills eval` takes a skill directory with an `evals/evals.json` file, generates benchmark tasks from it, runs them with and without the skill installed, and reports the "lift" — how much the skill improves agent performance. 0.6 task-standard validation is in [`docs/reports/2026-06-09-task-standard-validation.md`](./reports/2026-06-09-task-standard-validation.md). ## Quick start ### 1. Add evals to your skill ``` my-skill/ ├── SKILL.md ├── scripts/ │ └── helper.py └── evals/ # ← add this └── evals.json ``` ### 2. Write test cases ```json theme={null} { "version": "1", "skill_name": "my-skill", "defaults": { "timeout_sec": 300, "judge_model": "gemini-3.1-flash-lite" }, "cases": [ { "id": "test-001", "question": "Do X using the my-skill skill.", "ground_truth": "expected output", "expected_behavior": [ "Agent read the SKILL.md file", "Agent ran helper.py with correct arguments", "Agent produced the expected output" ] } ] } ``` ### 3. Run the eval ```bash theme={null} bench skills eval my-skill/ --agent claude-agent-acp ``` Expected output: ``` $ bench skills eval ./my-skill/ --agent claude-agent-acp Skill eval: my-skill (1 cases) Agents: claude-agent-acp Environment: docker Skill Eval: my-skill ┏━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━┓ ┃ Agent ┃ Mode ┃ Score ┃ Avg Reward ┃ ┡━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━┩ │ claude-agent-acp │ with-skill │ 1/1 │ 0.90 │ │ claude-agent-acp │ baseline │ 0/1 │ 0.20 │ │ claude-agent-acp │ LIFT │ +1 │ +0.70 │ └───────────────────┴────────────┴───────┴────────────┘ ``` ## evals.json reference ### Top-level fields | Field | Type | Required | Description | | -------------------------- | ------ | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | | `version` | string | No | Schema version (default: "1") | | `skill_name` | string | No | Skill name (auto-detected from SKILL.md) | | `defaults.timeout_sec` | int | No | Per-task timeout in seconds (default: 300) | | `defaults.judge_model` | string | No | Model for LLM judge (default: gemini-3.1-flash-lite) | | `defaults.skill_mount_dir` | string | No | Neutral sandbox path where the generated task exposes the skill before BenchFlow links it into agent-specific discovery paths (default: /skills) | ### Case fields | Field | Type | Required | Description | | ------------------- | --------- | -------- | ----------------------------------------------------- | | `id` | string | No | Unique case ID (auto-generated if missing) | | `question` | string | **Yes** | The task instruction sent to the agent | | `ground_truth` | string | No | Expected final answer (used for exact match fallback) | | `expected_behavior` | string\[] | No | Behavioral rubric for LLM judge | | `expected_skill` | string | No | Which skill should be invoked | | `expected_script` | string | No | Which script should be called | | `environment` | object | No | Per-case env var overrides | ### Grading logic * If `expected_behavior` is provided → **LLM judge** scores the agent's trajectory against the rubric (0.0-1.0) * If only `ground_truth` is provided → **exact match** checks if the answer appears in agent output (0.0 or 1.0) * If neither → reward is 0.0 ### Agent and judge credentials `bench skills eval` runs real agents. The selected agent must have whatever provider credentials or subscription auth it normally needs, and LLM-judge cases also need a supported judge key available in the environment. Exact-match cases can avoid the judge model, but they still need a working agent. For Codex agents, that auth can be `OPENAI_API_KEY`, `CODEX_API_KEY`, `CODEX_ACCESS_TOKEN`, or a host `~/.codex/auth.json` login. Provider-prefixed models can use provider-specific credentials instead; Azure Foundry models use `AZURE_API_KEY` plus `AZURE_API_ENDPOINT`. When a supported judge key is present on the host (`GOOGLE_API_KEY`, `GEMINI_API_KEY`, `ANTHROPIC_API_KEY`, or `OPENAI_API_KEY`), generated tasks reference it through `[verifier.env]` template syntax such as `${GEMINI_API_KEY}`. Secret values are resolved at verifier runtime and are not written into generated task files. The `oracle` agent is useful for generic task and sandbox smoke tests, but it is not a replacement for skill evaluation. Skill-eval tasks are generated from questions and rubrics and do not include `solution/solve.sh`, so oracle runs will error instead of measuring skill lift. ### Existing task-embedded skills Skills embedded under a benchmark task, such as `tasks//environment/skills//SKILL.md`, are task-local skill packs. They are not exposed to ordinary no-skills runs by default. To evaluate one directly with `bench skills eval`, add a sibling `evals/evals.json` inside that skill directory or copy the skill into a standalone skill directory with the same `evals/` contract. The repo includes a real standalone example at [`skills/citation-management/`](../skills/citation-management/), adapted from the SkillsBench `citation-check` task: ```bash theme={null} bench skills eval skills/citation-management \ --agent gemini \ --model gemini-3.1-flash-lite-preview \ --sandbox docker \ --jobs-dir jobs/skill-eval-citation-management \ --concurrency 1 ``` ## Multi-agent comparison Test your skill across multiple agents: ```bash theme={null} bench skills eval my-skill/ \ --agent claude-agent-acp --agent codex-acp --agent gemini ``` Expected output: ``` $ bench skills eval ./calculator/ --agent claude-agent-acp --agent codex-acp Skill eval: calculator (3 cases) Agents: claude-agent-acp, codex-acp Environment: docker Skill Eval: calculator ┏━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━┓ ┃ Agent ┃ Mode ┃ Score ┃ Avg Reward ┃ ┡━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━┩ │ claude-agent-acp │ with-skill │ 3/3 │ 0.95 │ │ claude-agent-acp │ baseline │ 1/3 │ 0.38 │ │ claude-agent-acp │ LIFT │ +2 │ +0.57 │ │ codex-acp │ with-skill │ 2/3 │ 0.72 │ │ codex-acp │ baseline │ 1/3 │ 0.35 │ │ codex-acp │ LIFT │ +1 │ +0.37 │ └───────────────────┴────────────┴───────┴────────────┘ ``` ## Custom environments For skills that need specific dependencies, add a Dockerfile: ``` my-skill/evals/ ├── evals.json ├── Dockerfile # custom container setup └── requirements.txt # extra Python deps ``` The Dockerfile is used instead of the default `python:3.12-slim` base. For with-skill runs, BenchFlow appends a `COPY skills/ /` step so the generated task exposes the skill at the neutral path declared in `task.toml`. During rollout setup, BenchFlow links that neutral path into the selected agent's configured discovery paths. ## GEPA integration Export traces for GEPA skill evolution: ```bash theme={null} bench skills eval my-skill/ --agent claude-agent-acp --export-gepa ``` This creates a GEPA-compatible export under `jobs/skill-eval//gepa/`: ``` jobs/skill-eval//gepa/ ├── skill.md # current SKILL.md content ├── traces/ # per-case execution traces with scores │ ├── test-001-claude-agent-acp-with.json │ └── test-001-claude-agent-acp-without.json └── summary.json # aggregate lift metrics ``` Feed these to GEPA to evolve your skill: ```python theme={null} import gepa optimizer = gepa.GEPA(traces_dir="traces/") improved_skill = optimizer.evolve("traces/skill.md") ``` ## End-to-End Walkthrough Here's a complete example evaluating a real skill from scratch. ### Step 1: Create the skill ```bash theme={null} mkdir -p gws-skill/scripts gws-skill/evals ``` Write `gws-skill/SKILL.md`: ```markdown theme={null} --- name: gws-email-drafting description: Draft professional emails using Gmail API patterns --- # GWS Email Drafting Use the templates in scripts/ to draft professional emails. ``` Write `gws-skill/scripts/draft_email.py`: ```python theme={null} import sys template = sys.argv[1] if len(sys.argv) > 1 else "general" print(f"Email drafted using {template} template") ``` ### Step 2: Write eval cases Write `gws-skill/evals/evals.json`: ```json theme={null} { "skill_name": "gws-email-drafting", "version": "1", "defaults": { "timeout_sec": 300, "judge_model": "claude-haiku-4-5-20251001" }, "cases": [ { "id": "draft-intro-email", "question": "Draft a professional introduction email to a potential workshop speaker. Use the gws-email-drafting skill.", "ground_truth": "The agent produced a professional email with subject line, greeting, body explaining the workshop, and call to action.", "expected_behavior": [ "The agent read the SKILL.md to understand the skill", "The agent used draft_email.py or followed the skill's patterns", "The email has a clear subject line", "The email body is professional and includes a call to action" ] }, { "id": "draft-followup", "question": "Draft a follow-up email to someone who hasn't responded in 2 weeks. Use the gws-email-drafting skill.", "ground_truth": "The agent produced a polite follow-up email that references the original outreach.", "expected_behavior": [ "The agent read the SKILL.md", "The email references a previous conversation", "The tone is polite but action-oriented", "The email is concise (under 200 words)" ] } ] } ``` ### Step 3: Run the eval ```bash theme={null} $ bench skills eval ./gws-skill/ --agent claude-agent-acp --agent codex-acp Skill eval: gws-email-drafting (2 cases) Agents: claude-agent-acp, codex-acp Environment: docker Skill Eval: gws-email-drafting ┏━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━┓ ┃ Agent ┃ Mode ┃ Score ┃ Avg Reward ┃ ┡━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━┩ │ claude-agent-acp │ with-skill │ 2/2 │ 0.92 │ │ claude-agent-acp │ baseline │ 1/2 │ 0.55 │ │ claude-agent-acp │ LIFT │ +1 │ +0.37 │ │ codex-acp │ with-skill │ 2/2 │ 0.88 │ │ codex-acp │ baseline │ 1/2 │ 0.48 │ │ codex-acp │ LIFT │ +1 │ +0.40 │ └───────────────────┴────────────┴───────┴────────────┘ ``` ### Step 4: Inspect results Results are saved to `jobs/skill-eval//`: ``` jobs/skill-eval/gws-email-drafting/ ├── claude-agent-acp/ │ ├── with-skill/ │ │ ├── draft-intro-email__abc123/ │ │ │ ├── result.json │ │ │ ├── trajectory/acp_trajectory.jsonl │ │ │ └── timing.json │ │ └── draft-followup__def456/ │ │ └── ... │ └── baseline/ │ └── ... └── codex-acp/ └── ... ``` ### Step 5: Improve with GEPA (optional) ```bash theme={null} $ bench skills eval ./gws-skill/ --agent claude-agent-acp --export-gepa GEPA traces exported to jobs/skill-eval/gws-email-drafting/gepa ``` Feed traces to the SkillSpin improvement pipeline to automatically evolve the skill text based on failure patterns. ## Architecture ``` ┌──────────────────────────────────────────────────────────────────┐ │ bench skills eval │ ├──────────────────────────────────────────────────────────────────┤ │ │ │ ┌─────────────┐ ┌──────────────────┐ ┌────────────────┐ │ │ │ evals.json │───▶│ Task Generator │───▶│ Ephemeral │ │ │ │ (2-8 cases) │ │ (with/without │ │ BenchFlow Tasks │ │ │ └─────────────┘ │ skill mode) │ │ (auto-deleted) │ │ │ └──────────────────┘ └───────┬────────┘ │ │ │ │ │ ┌─────────────┐ ┌──────────────────┐ ┌───────▼────────┐ │ │ │ Lift Report │◀───│ Result Collector │◀───│ Job Engine │ │ │ │ (per agent) │ │ (per case×mode) │ │ (concurrency, │ │ │ └─────────────┘ └──────────────────┘ │ retries, ACP) │ │ │ └────────────────┘ │ │ │ │ With-skill tasks bake the skill at /skills by default; │ │ BenchFlow links that neutral path into each agent's skill paths.│ │ │ │ ┌─────────────────────────────────────────────────────────┐ │ │ │ LLM Judge │ │ │ │ Reads: trajectory + case.json (ground_truth, rubric) │ │ │ │ Writes: /logs/verifier/reward.txt (0.0-1.0) │ │ │ └─────────────────────────────────────────────────────────┘ │ └──────────────────────────────────────────────────────────────────┘ ``` ## For Skill Developers (Jon Snow Adapter Pattern) If you maintain skills and want CI-integrated eval: ``` my-skill/ ├── SKILL.md ├── scripts/ │ └── do_something.py └── evals/ └── evals.json ← 2-4 test cases ``` That's it. No benchmark task authoring, no Dockerfiles, no test scripts. BenchFlow generates everything ephemeral — only results persist. **CI integration:** ```bash theme={null} # In your skill's CI pipeline uv tool install --prerelease allow benchflow # --prerelease allow is for the pinned LiteLLM rc dependency, not benchflow itself. bench skills eval . --agent claude-agent-acp --no-baseline ``` **What the adapter does (zero LLM):** ``` evals.json → Generate benchmark tasks → Run agents → Grade → Cleanup (static) (deterministic) (ACP) (LLM) (auto) ``` The adapter is purely deterministic — no LLM in task generation. LLM is only used at grading time (the judge). ## Tips for writing good eval cases 1. **Be specific in questions** — "Use the calculator skill to compute X" is better than "Compute X" 2. **Write 3-5 rubric items per case** — Each should be independently verifiable from the trajectory 3. **Include edge cases** — Test error handling, unusual inputs, multi-step workflows 4. **Keep ground\_truth simple** — Exact match works best for numeric or short-string answers 5. **Use 2-4 cases minimum** — Enough to show a pattern, not so many that runs get expensive 6. **Test the lift, not just correctness** — The goal is to show the skill improves performance vs baseline. If baseline already scores high, the skill isn't adding value # Task authoring Source: https://docs.benchflow.ai/task-authoring # Authoring tasks A BenchFlow task packages an instruction, a sandboxed environment, and a verifier into a directory that BenchFlow runs and scores automatically. This page covers the Harbor-compatible split layout (`task.toml` + `instruction.md`). For the native single-document format, see [Authoring native task.md tasks](./task-authoring-task-md.md). *** ## Directory layout > \[!NOTE] > BenchFlow will provide first-party support for hosted competition platforms, Verifiers, and OpenReward Standard. You can create [Harbor-format tasks](https://www.harborframework.com/docs/tasks) in BenchFlow with a `task.toml` config file, separate `instruction.md`, sandbox assets under `environment/`, verifier files under `tests/`, and an optional `solution/` oracle. ``` my-task/ ├── task.toml # timeouts, resources, metadata ├── instruction.md # what the agent must do ├── environment/ │ └── Dockerfile # sandbox image ├── tests/ │ └── test.sh # verifier entry point └── solution/ # optional — reference/oracle solution └── solve.sh ``` `tests/` may also include `test_outputs.py` (pytest module called by `test.sh`). *** ## task.toml ```toml theme={null} version = "1.0" [metadata] # optional, freeform author_name = "alice" difficulty = "easy" # easy / medium / hard category = "programming" tags = ["bash", "files"] [agent] timeout_sec = 300 # strongly recommended — unset means no wall-clock cap # user = "agent" # optional — run agent as this user/UID [verifier] timeout_sec = 120 # optional (default 600) [environment] cpus = 1 # default 1 memory_mb = 2048 # default 2048 storage_mb = 10240 # default 10240 allow_internet = false # default true env = { OPENAI_API_KEY = "${OPENAI_API_KEY}" } # host vars to inject ``` **Service-backed tasks** — BenchFlow ships a small service registry for task-local APIs such as Gmail, Slack, Calendar, Docs, and Drive. The runner does not auto-start services just because a Dockerfile references a binary. For Python-driven runs, start services explicitly with `pre_agent_hooks=build_service_hooks([...])`; for CLI-only task authoring, keep services inside the task's own Dockerfile/startup scripts until a dedicated service declaration is wired through the CLI. **Install tooling to shared prefixes, not `/root`** — when a task image ships Node.js, Python tools, or agent binaries that the sandbox user must execute, install them to `/usr/local/bin`, `/usr/local/lib`, or `/opt`, not `/root/.nvm` or `/root/.local/bin`. `setup_sandbox_user()` creates the non-root user, prepares small config/auth dirs, and chowns the workspace — it does not clone `/root` into the sandbox home. Legacy images that already install tools under `/root` still work via a narrow symlink fallback, but shared prefixes are the supported path. Pre-creating the sandbox user in the Dockerfile is an optional speedup, not a requirement. *** ## Multi-container tasks A task may ship an `environment/docker-compose.yaml` alongside the `Dockerfile`. The agent always runs in the `main` service; any additional services you declare become sibling containers on the same Docker network. This supports vulhub-style CVE tasks where the agent attacks a separate target container over the network. > `environment/Dockerfile` is always required — `bench tasks check` rejects > a task that ships only a `docker-compose.yaml`. If your `main` service > uses a prebuilt `image:` and needs no build context, still include a > minimal `Dockerfile` (e.g. `FROM `) so structural validation > and other tooling agree on the task package shape. ```yaml theme={null} # environment/docker-compose.yaml services: main: {} # agent container — BenchFlow injects build/image/limits target: # vulnerable service the agent must exploit image: vulhub/struts2-s2-001:latest expose: ["8080"] ``` `main` reaches `target` by service name (`http://target:8080`). The verifier can inspect *target-side* state — not just the agent's workspace — by passing a `service` argument when running commands: ```python theme={null} # In a Python-driven run or pre/post hook await env.exec_in_service("target", "test -f /tmp/exploit_proof.txt") await env.exec("cat /flag", service="target") # equivalent form services = await env.inner.services() # ["main", "target"] ``` `exec(..., service=...)` works on the Docker sandbox and the Daytona DinD (compose) sandbox. Single-container backends (Modal, direct Daytona) raise a clear error for any non-`main` service. This lets a verifier check write-based oracles (`/tmp/exploit.txt` in the target), database modifications, or RCE markers without trusting the agent container. ### Target-side `test.sh` verification For tasks whose success oracle lives in a target container — an RCE marker file, a modified database row — point the `test.sh` verifier at that service with `[verifier].service`: ```toml theme={null} [verifier] service = "target" # run tests/test.sh inside the `target` container ``` With this set, BenchFlow uploads the task's `tests/` directory into the **target** container, runs `test.sh` there, and copies the resulting `reward.txt` / `reward.json` back to the host. `service` defaults to `"main"` (the agent container), so existing single-container tasks are unaffected. `[verifier].service` is the declarative, task-schema way to do cross-container verification; the `env.exec_in_service(...)` Python API above is the imperative equivalent for hook-driven runs. > Use the same `service` name you declared in `docker-compose.yaml`. A > `test.sh` running in the target reaches `main` (and vice versa) by service > name over the Docker network, just like the agent does. ### Hardening policy for multi-container tasks BenchFlow's pre-verification hardening — killing the sandbox user's processes, scrubbing `PATH`/`PYTHONPATH`, restoring build-config files — applies **only to the `main` (agent) container**. Target containers are deliberately left unhardened: a vulhub-style target is *meant* to be vulnerable, the agent never has a shell inside it, and hardening it would risk breaking the very vulnerability the task exercises. `[verifier].service` selects where `test.sh` *runs*; it does not move hardening off `main`. *** ## instruction.md The first prompt sent to the agent. Write it as you would for a skilled developer: * State the precise goal in the first sentence. * Name exact files or paths the agent must create or modify. * Specify constraints (no external libraries, must pass existing tests, etc.). * Don't mention the verifier or `reward.txt` — those are internal. **Multi-turn prompts** — use a Scene with multiple Turns. A `None` prompt means "use `instruction.md`": ```python theme={null} from benchflow.rollout import RolloutConfig, Scene, Role, Turn config = RolloutConfig( task_path="tasks/my-task", scenes=[Scene( roles=[Role("agent", "gemini", "gemini-3.1-flash-lite-preview")], turns=[ Turn("agent"), # instruction.md Turn("agent", "Review your solution and fix any test failures."), ], )], environment="daytona", ) result = await bf.run(config) ``` *** ## Verifier contract (tests/test.sh) After the agent finishes, the BenchFlow runtime copies `tests/` to `/tests/` and runs `/tests/test.sh`. The working directory is the Dockerfile's `WORKDIR` (typically `/app/` in the example Dockerfile below). **Your script must write a single float (0.0–1.0) to `/logs/verifier/reward.txt`.** After writing the reward, exit `0`; a nonzero `test.sh` exit is treated as verifier infrastructure failure, not a scored task failure. | Path | Contents | | ----------------- | ---------------------------------------------------- | | `/app/` | Agent's working directory | | `/tests/` | Your `tests/` directory | | `/solution/` | `solution/` (oracle runs only) | | `/logs/verifier/` | Write `reward.txt` (and optionally `ctrf.json`) here | ### Pure bash verifier ```bash theme={null} #!/bin/bash REWARD=0 if [ -f /app/hello.txt ] && [ "$(cat /app/hello.txt | tr -d '\n')" = "Hello, world!" ]; then REWARD=1 fi echo "$REWARD" > /logs/verifier/reward.txt ``` ### pytest verifier ```bash theme={null} #!/bin/bash curl -LsSf https://astral.sh/uv/0.9.7/install.sh | sh source $HOME/.local/bin/env uvx \ --with pytest==8.4.1 \ --with pytest-json-ctrf==0.3.5 \ pytest --ctrf /logs/verifier/ctrf.json /tests/test_outputs.py -rA if [ $? -eq 0 ]; then echo 1; else echo 0; fi > /logs/verifier/reward.txt ``` ### Partial credit ```bash theme={null} python3 -c "print($PASSED / $TOTAL)" > /logs/verifier/reward.txt ``` **Security:** don't let the agent write to `/logs/verifier/reward.txt` or modify `/tests/test.sh`. For tasks running arbitrary code, use `allow_internet = false` and verify output files only. For LLM agent runs, BenchFlow preserves the network path needed for model APIs and agent startup, then disables supported agent web browsing/fetch tools through agent config or launch controls. Oracle runs still use the environment's network policy directly. *** ## solution/ (optional) Include when you want to verify the task is solvable or provide a reference implementation. When BenchFlow runs with `--agent oracle`, it copies `solution/` to `/solution/` and runs `solution/solve.sh` instead of an ACP agent. `solve.sh` has the same filesystem access as the agent — write only to `/app/`, not to `/logs/verifier/`. ```bash theme={null} #!/bin/bash echo "Hello, world!" > /app/hello.txt ``` *** ## CLI ```bash theme={null} # Scaffold a new task in this legacy split layout # (without --format legacy, init scaffolds the native task.md format) bench tasks init my-task --format legacy bench tasks init my-task --format legacy --no-pytest --no-solution # Generate tasks from agent traces (personal benchmark curation) bench tasks generate --from-local # from local Claude Code sessions bench tasks generate --from-file session.jsonl --dry-run # from a JSONL trace file bench tasks generate --from-hf opentraces-test --limit 50 # from a HuggingFace dataset bench tasks list-sources # list known HF trace datasets # Validate structure bench tasks check tasks/my-task/ # Confirm oracle gets reward = 1.0 bench eval create --tasks-dir tasks/my-task/ --agent oracle --sandbox docker # Run a real agent bench eval create --tasks-dir tasks/my-task/ --agent gemini --sandbox daytona # Run with task-local skills mounted bench eval create \ --tasks-dir tasks/my-task/ \ --agent gemini \ --sandbox daytona \ --skill-mode with-skill \ --agent-env BENCHFLOW_SKILL_NUDGE=name ``` Task-local skills are mounted through the selected agent's native skill paths. See [Architecture: skill loading](./architecture.md#skill-loading) for the canonical loading semantics and nudge modes. `bench tasks generate` converts agent traces (Claude Code sessions, opentraces records, or HuggingFace datasets) into task directories with `task.toml`, `instruction.md`, and a file-existence `test.sh`. Use `--dry-run` to preview traces before generating. See [CLI reference](./reference/cli.md#bench-tasks-generate) for all flags. `bench tasks check` validates task definition presence (`task.md` or legacy `task.toml` + `instruction.md`), a non-empty instruction, `environment/Dockerfile`, and a runnable verifier entrypoint (`verifier/` or legacy `tests/`). It surfaces `task.toml` parse errors but does not require `[agent].timeout_sec` (unset means no wall-clock cap). Exits with code 1 on failure (CI-friendly). *** ## Worked example — write-fizzbuzz ```toml theme={null} # task.toml version = "1.0" [metadata] difficulty = "easy" tags = ["python"] [agent] timeout_sec = 180 [verifier] timeout_sec = 60 ``` ```markdown theme={null} # instruction.md Write a file `fizzbuzz.py` defining: def fizzbuzz(n: int) -> str Return "FizzBuzz" / "Fizz" / "Buzz" / str(n) for divisibility by 15 / 3 / 5 / none. No __main__ block, no print statements. ``` ```dockerfile theme={null} # environment/Dockerfile FROM ubuntu:24.04 RUN apt-get update -qq && apt-get install -y -qq python3 curl && rm -rf /var/lib/apt/lists/* WORKDIR /app RUN mkdir -p /logs/verifier /logs/agent /logs/artifacts ``` ```python theme={null} # tests/test_outputs.py import importlib.util from pathlib import Path def _load(): path = Path("/app/fizzbuzz.py") assert path.exists() spec = importlib.util.spec_from_file_location("fizzbuzz", path) mod = importlib.util.module_from_spec(spec) spec.loader.exec_module(mod) return mod.fizzbuzz def test_fizz(): assert _load()(3) == "Fizz" def test_buzz(): assert _load()(5) == "Buzz" def test_fizzbuzz():assert _load()(15) == "FizzBuzz" def test_number(): assert _load()(7) == "7" ``` ```bash theme={null} # tests/test.sh — verifier entrypoint; runs the pytest module, writes the reward #!/bin/bash curl -LsSf https://astral.sh/uv/0.9.7/install.sh | sh source $HOME/.local/bin/env uvx --with pytest==8.4.1 pytest /tests/test_outputs.py -rA if [ $? -eq 0 ]; then echo 1; else echo 0; fi > /logs/verifier/reward.txt ``` ```bash theme={null} # solution/solve.sh cat > /app/fizzbuzz.py << 'EOF' def fizzbuzz(n: int) -> str: if n % 15 == 0: return "FizzBuzz" if n % 3 == 0: return "Fizz" if n % 5 == 0: return "Buzz" return str(n) EOF ``` # Task authoring task md Source: https://docs.benchflow.ai/task-authoring-task-md # Authoring native task.md tasks A native BenchFlow task is one `task.md` document plus sidecar directories. The YAML frontmatter carries the task configuration; the markdown body **is** the prompt. This page teaches the native format hands-on. For the normative standard see [the task standard](./task-standard.md); for the legacy split layout (`task.toml` + `instruction.md` + `tests/` + `solution/`) see [Authoring tasks](./task-authoring.md). When a directory contains both layouts, `task.md` is the authoritative task definition — the runtime selects it and ignores the split pair. *** ## Minimal task — three files ```text theme={null} my-task/ ├── task.md # config frontmatter + prompt body ├── environment/ │ └── Dockerfile # sandbox image └── verifier/ └── test.sh # verifier entry point ``` That is the complete runnable surface: structural validation requires a task definition (`task.md`, or the legacy `task.toml` + `instruction.md` pair), an `environment/` directory with a `Dockerfile`, and a verifier directory with a runnable entrypoint. An `oracle/` directory is optional. ```markdown theme={null} --- agent: timeout_sec: 300 # strongly recommended — unset means no wall-clock cap verifier: timeout_sec: 120 environment: cpus: 1 memory_mb: 2048 --- Create a file `/app/hello.txt` containing exactly `Hello, world!`. ``` ```bash theme={null} #!/bin/bash # verifier/test.sh REWARD=0 if [ "$(cat /app/hello.txt 2>/dev/null | tr -d '\n')" = "Hello, world!" ]; then REWARD=1 fi echo "$REWARD" > /logs/verifier/reward.txt ``` Scaffold this shape with the CLI (task.md is the default format): ```bash theme={null} bench tasks init my-task # task.md, environment/, verifier/, oracle/ bench tasks check tasks/my-task # structural validation bench tasks check tasks/my-task --level schema # frontmatter + prompt parse only ``` *** ## Frontmatter `task.md` must start with a `---`-delimited YAML frontmatter block, and the frontmatter must be a mapping — a document without it fails to parse. The keys fall into three classes. **Task config keys** are the Harbor-compatible config surface, validated as `TaskConfig`. Unknown keys are **rejected** (the schema is `extra="forbid"`), so typos fail at parse time instead of becoming silently-ignored config: | Key | Meaning | | ---------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- | | `schema_version` (alias `version`) | Config schema version, currently `"1.3"` | | `task` | Package identity: `name` (`org/name` format), `description`, `authors`, `keywords` | | `metadata` | Freeform mapping — difficulty, category, tags, anything descriptive | | `agent` | Agent run policy: `timeout_sec`, `user`, `network_mode`, `allowed_hosts` | | `verifier` | Verifier run policy: `timeout_sec` (default 600), `env`, `user`, `service`, … | | `environment` | Sandbox: `docker_image`, `cpus`, `memory_mb`, `storage_mb`, `network_mode`, `env`, `workdir`, … | | `oracle` | Oracle run policy: `env`, `timeout_sec` (import alias: `solution`) | | `source`, `artifacts`, `steps`, `multi_step_reward_strategy`, `reward` | Provenance and Harbor-compatible extras | `agent.timeout_sec` is **strongly recommended**: it is optional and defaults to unset, and a task that omits it runs the agent with no wall-clock cap unless the caller supplies a per-run timeout. Set it on every published task. Declaring both `oracle` and the legacy `solution` alias in one config is invalid and rejected; native tasks use `oracle`. **Document orchestration keys** are parsed by `TaskDocument`, not `TaskConfig`: `agents` (named roles with `agent`, `model`, `reasoning_effort`, `capabilities`, …), `scenes` (ordered turns referencing declared roles — a turn that names an undeclared role is a parse error), and `user` (simulated user). `benchflow` is the reserved extension namespace. **Authoring shorthands** are expanded during parsing and never reach the canonical config under their short names: | Shorthand | Expands to | | ----------------------------------- | ---------------------------------------------------------------------- | | `name: hello-world` | `task.name: benchflow/hello-world` (a `/` in the value keeps your org) | | `image: ubuntu:24.04` | `environment.docker_image: ubuntu:24.04` | | `verifier: verifier/` (string form) | `benchflow.verifier.path` / `.spec` / `.entrypoint` defaults | | `oracle: oracle/` (string form) | `benchflow.oracle.path` | | `profile: code-change` | Merges a named defaults bundle (see below) | Profiles (`profile:` / `profiles:`) merge predefined default bundles — `code-change`, `harbor-compatible`, `reward-kit`, `acceptance-live`, `multi-agent`, `leaderboard-local` — under your explicit keys; an unknown profile name is a parse error. `bench tasks normalize ` prints the fully expanded canonical document (`--write` replaces `task.md` in place), so a minimal authored file and its canonical form never drift apart. *** ## Prompt body and prompts/ sidecars The body below the frontmatter is the base prompt — free-form markdown, no heading ceremony required. If the body contains no reserved section headings, the entire body is the instruction the agent receives. Four reserved headings are recognized for compatibility imports: `## prompt`, `## role:`, `## scene:`, and `## user-persona`. Repeating the same section heading is a parse error. `bench tasks init` scaffolds a single `## prompt` section as a starting point — for a single-prompt task that is equivalent to a bare body, so keep it or drop the heading as you prefer. The multi-prompt headings (`## role:`, `## scene:`, `## user-persona`) are for compatibility imports only; new multi-prompt material belongs in sidecar files under `prompts/`: | File | Meaning | | ------------------------- | ---------------------------------------------------- | | `prompts/role..md` | Role prompt — the whole file body is the prompt text | | `prompts/scene..md` | Scene prompt | | `prompts/user-persona.md` | Simulated-user persona | Sidecar files take precedence over a reserved heading of the same name, so a compat-imported task can be cleaned up incrementally. Runtime prompt precedence for a turn is: inline turn prompt, then scene prompt, then role prompt, then base prompt. A multi-role task wires the pieces together in frontmatter: ```yaml theme={null} agents: roles: solver: agent: claude-agent-acp scenes: - name: solve turns: - role: solver ``` with the solver guidance, if any, in `prompts/role.solver.md`. See [docs/examples/task-md/](./examples/task-md/README.md) for runnable examples, including real converted SkillsBench packages. *** ## Verifier package and strategy declaration The native verifier directory is `verifier/` (`tests/` remains the legacy alias; when both exist, `verifier/` wins and there is no fallback to `tests/`). At verify time the directory is uploaded into the sandbox at `/verifier` (legacy `tests/` uploads to `/tests`), and the verifier must write its reward to `/logs/verifier/reward.txt` (and optionally `/logs/verifier/reward.json`). A plain `verifier/test.sh` is a complete verifier: with no other declaration, the runtime executes it directly. The same contract as the legacy layout applies — write a float `0.0`–`1.0` to `/logs/verifier/reward.txt`, then exit `0`; a nonzero exit means verifier infrastructure failure, not a scored task failure. To declare *how* the task is scored, add `verifier/verifier.md`. Its frontmatter must contain a `verifier:` mapping with at least one entry under `strategies`; `default_strategy` selects which one runs (it defaults to the first declared strategy and must name a declared one): ```markdown theme={null} --- document_version: "0.3" verifier: name: my-task-verifier default_strategy: deterministic strategies: deterministic: type: script command: ./test.sh outputs: reward_text: /logs/verifier/reward.txt reward_json: /logs/verifier/reward.json --- ## verifier intent What the verifier measures and which task outputs it reads. ``` Five strategy types are recognized, each with fail-closed required fields: | `type` | Required config | Notes | | ------------- | -------------------------------------------- | ------------------------------------------------------------------------------------------------------ | | `script` | `command` | Runs as `cd /verifier && `; local script files named in the command must exist in `verifier/` | | `llm-judge` | `rubric` | Optional `model`, `input_dir`, and `context` *or* `context_file` (not both) | | `reward-kit` | `root` | Optional `entrypoint` (default `reward.py`) and `criteria`; paths must be safe-relative | | `agent-judge` | `role`, `isolation: verifier-only`, `inputs` | `role` must match a `## role:` section in the verifier.md body | | `ors-episode` | `inputs` | Optional `format`: `json`, `jsonl`, or `auto` | An unknown `type` is a parse error. `bench tasks check` also verifies the selected strategy is actually runnable — e.g. a `script` strategy whose referenced files are missing, or an `llm-judge` strategy whose rubric file does not exist, fails validation. `outputs` declares the reward artifact contract (defaults shown above; `details_json` and `aggregate_policy` are optional). `bench tasks check --level publication-grade` additionally requires the native package shape: `task.md`, native `oracle/`, `verifier/verifier.md` with rubric files, and an explicit `reward_json` output contract. *** ## Oracle `oracle/solve.sh` is the held-out reference solution (`solution/` is the legacy alias; `oracle/` wins when both exist). Native oracles are uploaded to `/oracle` in the sandbox (legacy `solution/` to `/solution`) and run instead of an agent with `--agent oracle`: ```bash theme={null} bench eval create --tasks-dir tasks/my-task --agent oracle --sandbox docker ``` A correct task scores `1.0` on its oracle run before any model sees it. *** ## Migrating a legacy task `bench tasks migrate` converts a `task.toml` + `instruction.md` pair into `task.md`: ```bash theme={null} bench tasks migrate tasks/my-task # writes task.md, keeps legacy files bench tasks migrate tasks/my-task --overwrite # replace an existing task.md bench tasks migrate tasks/my-task --remove-legacy # delete the split pair and # promote tests/ -> verifier/, # solution/ -> oracle/ ``` The migration is non-destructive by default and refuses to write anything lossy: the generated document is re-parsed and must reproduce the original config semantics and instruction text exactly, or the command fails. Unknown `task.toml` keys that the schema does not model are preserved under `benchflow.compat` in the generated frontmatter rather than dropped. After migrating, validate the result: ```bash theme={null} bench tasks check tasks/my-task bench eval create --tasks-dir tasks/my-task --agent oracle --sandbox docker ``` ## Exporting to the split layout To go the other way — produce a Harbor/Pier-compatible split layout from a `task.md` package — use `bench tasks export`: ```bash theme={null} bench tasks export tasks/my-task out/my-task-split # harbor target bench tasks export tasks/my-task --report-only # loss report only ``` The export writes a compatibility loss report to `compatibility/export-report.json` so you can see what (if anything) the split layout cannot represent. Publication-grade validation requires `task.md` to be the only authoritative entrypoint, so keep exported split layouts in a separate output directory rather than beside `task.md`. See [CLI reference: bench tasks export](./reference/cli.md#bench-tasks-export) for all flags. # Task standard Source: https://docs.benchflow.ai/task-standard # BenchFlow Task Package Standard Status: v0.6.2 — current stable task package standard (2026-06-14) This document defines the direction for BenchFlow-native task packages. The short version: `task.md` is the native authoring entrypoint; `oracle/` and `verifier/` are the BenchFlow-native names for held-out reference behavior and reward checks. Split-layout names such as `solution/` and `tests/` remain compatibility names for migration and export only. The standard is intentionally split into three views: | View | Purpose | Owner | | -------------------- | ----------------------------------------------------------------------- | --------------------------------------------------- | | Authoring document | What humans and generators write: one `task.md` plus sidecar dirs | `TaskDocument` | | Runtime task view | What rollout, verifier, hardening, provenance, and trajectories consume | `TaskRuntimeView` plus first `TaskPackage` boundary | | Foreign adapter view | What external benchmark formats and hosted environments import/export | adapters | Do not treat these as the same interface. A good authoring document can include more information than a foreign format can export, and a foreign import can preserve unknown data that native authoring would reject. ## Goals and scope A BenchFlow task is one `task.md` that selects a mode on each of three planes (see *Planes* below): how the environment is built, how the agent interacts, and how the result is scored. The standard has three goals: 1. **Native authoring** — humans and generators write one `task.md` (plus sidecar dirs) as the primary surface. 2. **Interoperability** — existing split-layout formats (`task.toml` + `instruction.md` + `solution/` + `tests/`) import directly, and packages export back to that layout with an explicit, honest loss report when a native concept has no equivalent. 3. **Coverage** — the schema can express the full range of eval shapes (single-shot, multi-round, simulated-user, multi-agent, and live-arena interaction; workspace, trajectory, rubric, judge, and leaderboard reward), even where the runtime for a given mode lands in a later milestone (see *Open Primitives and Roadmap*). Native authoring is the priority surface; import is best-effort-faithful; export is best-effort with a loss report. The standard does not require bidirectional-lossless round-tripping with any one external format. ## Field discipline Every normative standard field must map to a mode on one of the three planes (below) or to a concrete BenchFlow runtime need. Fields without that mapping belong under `metadata` or in an adapter, not in the standard. ## Native Layout ```text theme={null} task/ |-- task.md |-- environment/ | `-- Dockerfile |-- verifier/ | |-- verifier.md | `-- test.sh |-- oracle/ | `-- solve.sh `-- evidence/ `-- validation.json ``` Compatibility aliases: | Native | Compatibility / foreign export | Meaning | | ----------- | ------------------------------ | ----------------------------------------------------------------- | | `task.md` | `task.toml` + `instruction.md` | Task config plus prompt material | | `oracle/` | `solution/` | Held-out reference implementation | | `verifier/` | `tests/` | Verifier package, reward code, hidden checks, rubrics, and judges | Target validation: native packages may carry both native and compatibility directories only when hashes prove the duplicate content is equivalent. Current runtime prefers the native spelling when both aliases exist; it does not yet prove equivalence. Within a BenchFlow-native package, selection is fail-closed: * if `task.md` exists, it is the authoritative task definition * if `verifier/` exists, it is the authoritative verifier directory; an empty or invalid `verifier/` does not fall back to `tests/` * if `oracle/` exists, it is the authoritative oracle directory; an empty or invalid `oracle/` does not fall back to `solution/` * duplicate alias trees must be byte-identical after normalized traversal or validation should report a collision As of v0.6.2, split layouts are compatibility inputs and export artifacts, not the native authoring surface. New BenchFlow tasks should publish `task.md` as the only authoritative entrypoint. ## Versioning There are two versions: ```yaml theme={null} schema_version: "1.3" # BenchFlow task config surface benchflow: document_version: "0.6" # BenchFlow task.md document syntax ``` `schema_version` is for the runtime config model shared with compatibility imports. `benchflow.document_version` is for document-only concepts such as teams, prompt composition, agent policy, runtime policy, private assets, provenance, evidence, nudges, and export policy. ## Root Frontmatter The root frontmatter has three classes of keys. BenchFlow config keys are modeled by `TaskConfig` and must be rejected when unknown in native authoring mode: * `schema_version` / `version` * `task` * `metadata` * `agent` * `verifier` * `environment` * `oracle` (validation alias: `solution`) * `source` * `artifacts` * `steps` * `multi_step_reward_strategy` Document orchestration keys are parsed by `TaskDocument`: * `agents` * `scenes` * `user` BenchFlow extension keys live under the reserved namespace: * `benchflow` Do not add new root keys for every new idea. Put draft or BenchFlow-specific extensions under `benchflow:` until they have a stable interface. Native authoring should use `oracle`. Importers may accept the compatibility `solution` alias, but a config that contains both names is invalid. ## Prompt Body The `task.md` body **is** the base prompt — free-form markdown, exactly like a `SKILL.md` body. No `## prompt` heading is required: if the body carries no reserved section headings, the entire body is the prompt. The common single-shot task is just frontmatter plus prose, so a bespoke benchmark ports by dropping its existing instruction text in as the body with no markup. Tasks that need more than a base prompt — multiple roles, multiple scenes, or a simulated-user persona — author each as its **own** free-form file under `prompts/`, so no body ever carries reserved-heading ceremony: | File | Meaning | | ------------------------- | ---------------------- | | `prompts/role..md` | Role prompt | | `prompts/scene..md` | Scene prompt | | `prompts/user-persona.md` | Simulated user persona | Each sidecar file is itself a clean free-form body. For backward compatibility, single-prompt source formats may instead embed the same content in the body via reserved `## prompt`, `## role:`, `## scene:`, and `## user-persona` headings; these import losslessly and normalize to the file layout. Sidecar files take precedence over a heading of the same name. New tasks should prefer the files. Default runtime precedence remains a simple fallback: 1. inline turn prompt 2. scene prompt 3. role prompt 4. base prompt Native tasks can make composition explicit: ```yaml theme={null} benchflow: prompt: composition: append order: [base, role, scene, turn] ``` The first `TaskPackage` prompt plan now compiles `append` and explicit `replace` policies deterministically. That avoids losing role guardrails when a scene prompt is present. `RolloutConfig` consumes the compiled plan for task.md scene execution when explicit CLI/SDK prompts do not override the document. ## Agent And Runtime Policy Agent isolation is distinct from task environment networking. A no-search task can still need dependency, LLM, or provider egress outside the sandbox; a no-network task can still need the agent harness to call its model. ```yaml theme={null} benchflow: agent_policy: skill_access: none # none | installed | declared allowed_skill_roots: [] search: disabled # allowed | disabled forbidden_tools: [web_search, browser_fetch] trajectory_audit: require_no_forbidden_tool_calls: true runtime_policy: backend: modal # docker | daytona | modal | kubernetes | podman | hpc | queue required_capabilities: [gpu:B200, private_mounts, persistent_state] network: task_default: no-network agent_egress: model-and-dependencies allowed_hosts: [] phase_overrides: verifier: no-network private_mounts: - source: modal-volume://org-models/qwen target: /mnt/models/qwen mode: ro visibility: agent secret_ref: org-models-readonly persistent_state: required: true scope: task-run cleanup: after-verifier ``` Unsupported `agent_policy`, `runtime_policy`, private mounts, registry secrets, GPU types, phase-specific network overrides, or persistent state semantics must fail closed before launch for the selected sandbox. No-search proof must come from both launch policy and trajectory audit; it is not implied by `network_mode: no-network`. ## Planes The package has six planes. Keeping them separate is the core abstraction. | Plane | Owns | Should not own | | ----------- | ---------------------------------------------------------------------------------------------------------------------------- | -------------------- | | Package | identity, versions, provenance, source hashes, export policy | sandbox execution | | Runtime | environment image, resources, network policy, setup/reset/readiness, mounts | prompt orchestration | | Interaction | agents, roles, teams, scenes, turns, user/nudge loops, handoff policy | verifier checkpoints | | Verifier | verifier package entrypoint, reward file contract, hidden fixtures, scorer type, rubrics, judge roles, separate verifier env | agent prompt text | | Oracle | held-out reference implementation and oracle-only env | tests/verifier code | | Evidence | validation runs, flake data, anti-cheat review, artifact hashes, leaderboard metadata | mutable task source | Imported `steps` are verifier/runtime checkpoints. BenchFlow `scenes` are interaction checkpoints. They are orthogonal. A task can have both, but a runtime must define how they compose before executing them. Of the six planes, three are the **authoring axes** an author selects a mode from — Runtime (the environment), Interaction, and Verifier — while the remaining three (Package, Oracle, Evidence) are supporting planes. A task is one mode-selection per axis: `environment` × `interaction-mode` × `verifier-strategy`. New normative fields must add or compose a mode on one of these axes; otherwise they belong in `metadata` or an adapter. ## Presets And Profiles Two different reuse mechanisms were historically both called "profile," which is why the mental model felt overloaded. v0.6 separates them: * **`preset`** — *authoring sugar*. A named bundle of defaults that `bench tasks normalize` expands into the canonical contract and then **discards**; it has no runtime existence. Presets are the lightweight-authoring path: a tiny `task.md` becomes a full contract. Examples: `code-change`, `acceptance-live`, `harbor-compatible`. (The current implementation still spells the preset key as `profile:`; M0 renames it to `preset:` and keeps `profile:` as a deprecated alias.) * **`*-profile`** — *runtime-real shared config*. A benchmark-level object that many tasks **reference** and that **persists into `TaskRuntimeView`**, one per authoring axis: * `environment-profile` — a shared world reused across tasks (e.g. one clawsbench service catalog referenced by all 44 tasks) * `agent-profile` — shared harness / model / role wiring * `verifier-profile` — shared reward strategy and rubric Rule of thumb: if removing it changes only how much you typed, it is a `preset`; if removing it changes what runs, it is a `*-profile`. Presets normalize away; profiles are inherited. `*-profile` resolution is M1 runtime work; `preset` exists today. ## Verifier Package The verifier is a peer package, not just a `tests/` directory. Native verifier packages should have their own entry document: ```text theme={null} verifier/ |-- verifier.md |-- task.toml |-- test.sh |-- rewards/ | |-- correctness/ | | `-- reward.py | `-- quality/ | `-- criteria.toml |-- rubrics/ | |-- verifier.md | `-- verifier.toml |-- judges/ | `-- reviewer.md `-- fixtures/ `-- hidden_cases.jsonl ``` `verifier/verifier.md` is analogous to `task.md` for the evaluation side. It describes what evidence is read, which strategies can score the task, how rubric dimensions compose, which judge agents are allowed, and what outputs the runtime must preserve. `verifier/task.toml` is an optional compatibility projection of `verifier/verifier.md`; it is not a second native surface. If both files exist, their canonical projection must match or validation fails closed. ```md theme={null} --- document_version: "0.3" verifier: name: hidden-patch-and-quality default_strategy: deterministic strategies: deterministic: type: script command: ./test.sh rewardkit: type: reward-kit root: rewards/ criteria: rewards/quality/criteria.toml judge: type: agent-judge role: verifier_judge model: gpt-5.5 inputs: [trajectory/acp_trajectory.jsonl, /logs/artifacts/patch.diff] isolation: verifier-only ors: type: ors-episode reward_aggregation: last_non_null terminal_policy: finished_true rubric: combine: weighted_sum dimensions: correctness: {weight: 0.7, source: deterministic} maintainability: {weight: 0.2, source: judge} evidence_quality: {weight: 0.1, source: rewardkit} files: human: rubrics/verifier.md structured: rubrics/verifier.toml outputs: reward_text: /logs/verifier/reward.txt reward_json: /logs/verifier/reward.json details_json: /logs/verifier/reward-details.json aggregate_policy: field: reward fallback: weighted_mean --- ## role:verifier_judge Grade only the submitted artifact and declared evidence. Do not infer intent from private oracle files or hidden verifier fixtures. ``` `test.sh` is the minimum executable strategy and the compatibility export target. Reward Kit-style criteria, ORS-style episode rewards, and AgentBeats-style assessor agents are verifier strategies. Runtime adapters may write declared evidence artifacts such as `trajectory/ors-rewards.jsonl`, but the ORS-specific judge/normalization semantics stay inside verifier scope. They must not leak into agent prompts, and they must emit either the canonical reward envelope or a declared multi-metric map. Verifier packages must be isolated: * judge models and credentials are verifier-scoped, not agent-visible * judge prompts live under `verifier/`, not inside the task prompt * rubrics are separate from judge persona: `rubrics/verifier.md` is the human scoring contract, `rubrics/verifier.toml` or JSON is the structured scoring contract, and `## role:verifier_judge` is the assessor posture * hidden fixtures and rubrics are mounted only during verifier execution * agent-as-judge trajectories are recorded as evidence and audited for input leakage * unsupported verifier strategies fail closed before scoring starts ## Verifier Contract Native verifier code lives in `verifier/` and is mounted at `/verifier`. Compatibility `tests/` remains supported and is mounted at `/tests`. The minimum script verifier contract applies to the selected verifier entrypoint: native packages with no `verifier/verifier.md` run `verifier/test.sh`; imported split-layout packages may run `tests/test.sh`. When `verifier/verifier.md` is present, structural verifier validity follows the selected strategy. A package can therefore be valid without `verifier/test.sh` if, for example, the selected Reward Kit runner and criteria files exist. * write `/logs/verifier/reward.txt` with one float from `0.0` to `1.0` * optionally write `/logs/verifier/reward.json` with structured rubric/evidence; BenchFlow preserves structured rewards, keeps `reward.txt` as scalar compatibility, and can compute a scalar from declared aggregate policies * prefer exit `0` after writing reward * treat nonzero verifier exit without a fresh reward file as infrastructure failure; a nonzero exit with a fresh reward is a scored task result with verifier diagnostics The richer standard should model verifier inputs explicitly: ```yaml theme={null} verifier: type: test-script timeout_sec: 900 benchflow: verifier: entrypoint: verifier/test.sh visibility: hidden inputs: - path: verifier/test.patch kind: test_patch visibility: hidden_verifier outputs: reward_text: /logs/verifier/reward.txt reward_json: /logs/verifier/reward.json transfer: agent_to_verifier: - /logs/artifacts/** verifier_to_result: - /logs/verifier/reward.* - /logs/verifier/artifacts/** ``` This covers `tests/test.patch`, Windows entrypoints, artifact-only graders, separate verifier images, and hidden fixtures without overloading the directory name. Separate verifier execution must define image resolution, hidden fixture mounting, step-level verifier environments, and transfer rules. Hidden verifier inputs such as `tests/test.patch` or `verifier/test.patch` are mounted only for the verifier phase; agent logs and workspace files move into verifier scope only through declared artifact transfer paths. Target reward precedence: * `reward.txt` remains the scalar compatibility minimum * when `reward.json` exists, it should be the authoritative rich reward artifact * `reward.json` may be an envelope with a numeric `reward`, or a reward-kit style multi-metric map with `metrics` plus a declared `aggregate` policy * if both files exist and `reward.json` has a scalar aggregate, the scalar must match `float(reward.txt)` or validation should fail closed * if `reward.json` is a multi-metric map without a scalar `reward`, verifier metadata may declare the aggregate policy used for scalar exports; current runtime computes `reward` for `mean`, `weighted_mean`, and `weighted_sum`, and `reward.txt` may still carry the scalar compatibility value * `reward-details.json` should be preserved when present * reward artifacts should preserve structured reserved keys such as `rubric`, `items`, `evidence`, `artifacts`, `metadata`, `reason`, `reasons`, `errors`, and task-specific payloads such as `metrics`, `regressions`, `participants`, `winner`, `raw`, and `debug` Current runtime now prefers `reward.json` over `reward.txt`, rejects disagreeing scalar outputs, and preserves a first set of structured reward fields. `src/benchflow/task/verifier_document.py` parses `verifier/verifier.md` strategies, rubric metadata, output contracts, and verifier-scoped role prompts. When `verifier/verifier.md` is present, `Verifier.verify()` selects its default strategy: `script` runs the declared command relative to the uploaded verifier directory, `llm-judge` uses the existing deliverables judge with verifier-local rubric, model, input directory, and context/context-file overrides, `reward-kit` runs a safe relative `reward.py` package runner inside verifier scope, writes a `reward-kit-manifest.json` contract, and, when criteria are declared, parses those criteria before launch and computes/verifies canonical `reward` from matching `reward.json.metrics`. `agent-judge` runs a verifier-scoped judge role over declared evidence inputs. ORS runtime helpers can normalize tool-output rewards into `trajectory/ors-rewards.jsonl`; `ors-episode` then reads declared ORS reward evidence, normalizes reward responses or event streams through the existing ORS adapter, and emits canonical `reward.json` plus `reward-details.json`. `reward-details.json` is a named rollout artifact, stale copies are cleared before verification, script verifiers can preserve it, target-service verifiers download it with the rest of `/logs/verifier`, and the built-in LLM judge emits criterion details there. Metrics-only `reward.json` maps can now use the verifier document's `outputs.aggregate_policy` or selected Reward Kit criteria policy to compute and persist the canonical `reward`. It still does not fully match the target: full Reward Kit parity, full OpenReward environment import/export, and AgentBeats assessor lifecycles are not yet first-class; selected unsupported strategies fail closed. Validation evidence should include parser checks, legacy-vs-native migration parity, live rollout artifacts, negative-contract failures, and explicit fail-closed results for parsed fields that the selected runtime cannot honor. ## Oracle Contract Native oracle code lives in `oracle/` and is mounted at `/oracle`. Compatibility `solution/` remains supported and is mounted at `/solution`. The naming change is semantic: * `oracle` means "held-out reference behavior used to prove solvability" * `solution` is a compatibility export name for older split layouts Proposed outbound exporters should map native `oracle/` to foreign `solution/`. Current runtime supports native and compatibility oracle paths, and the first split-layout exporter maps native `oracle/` to `solution/`. Publication-grade oracle equivalence enforcement remains target behavior. ## Assets, Provenance, And Evidence Native task packages need more than source code. The standard should track assets as first-class objects: ```yaml theme={null} benchflow: provenance: images: - field: environment.docker_image reference: ghcr.io/org/task-image:2026-06 digest: sha256:... registry: ghcr.io provider: modal build_source: path: environment/Dockerfile sha256: "" credential_secret_ref: task-image-pull never_persist_credentials: true assets: - path: environment/dataset.parquet visibility: agent sha256: "" source: url: https://huggingface.co/datasets/org/name revision: "" license: apache-2.0 generated: false mount_phase: agent - path: verifier/hidden_cases.jsonl visibility: hidden_verifier sha256: "" mount_phase: verifier secrets: - name: task-image-pull scope: image-pull visibility: runtime_secret never_persist: true ``` Suggested visibility values: * `agent` * `hidden_verifier` * `hidden_oracle` * `runtime_secret` * `external_dataset` * `evidence_only` Evidence should prove task validity, not just package shape: ```yaml theme={null} benchflow: evidence: oracle_runs: required_reward: 1.0 last_job: jobs/task-standard/oracle/... artifact: evidence/calibration/oracle-run.json verifier: reruns: 5 flake_rate: 0.0 report: evidence/calibration/verifier-stability-report.json review: anti_cheat: passed instruction_alignment: passed reviewer: benchflow artifact: evidence/calibration/review.json calibration: no_op_reward_max: 0.0 known_bad_reward_max: 0.2 partial_solution_range: [0.3, 0.8] report: evidence/calibration/calibration-report.json human_or_reference_examples: - name: gold expected_reward: 1.0 artifact: evidence/calibration/gold-result.json judge_agreement: required: true sample_count: 5 min_pairwise_agreement: 0.8 trajectories: - path: trajectory/acp_trajectory.jsonl kind: acp visibility: evidence_only sha256: "" - path: trajectory/critique.jsonl kind: critique visibility: evidence_only sha256: "" artifacts: - path: evidence/calibration/oracle-run.json kind: oracle_run visibility: evidence_only sha256: "" - path: evidence/calibration/verifier-stability-report.json kind: verifier_stability visibility: evidence_only sha256: "" - path: evidence/calibration/review.json kind: acceptance_review visibility: evidence_only sha256: "" - path: evidence/calibration/calibration-report.json kind: calibration_report visibility: evidence_only sha256: "" - path: evidence/calibration/gold-result.json kind: reference_result visibility: evidence_only sha256: "" - path: artifacts/session.har kind: har mime_type: application/json produced_by: browser visibility: evidence_only sha256: "" ``` Borrow the discipline, not the package shape, from METR-style task standards: * keep BenchFlow document-first rather than Python `TaskFamily`-first * make asset visibility explicit instead of relying on directory folklore * declare required secrets/resources with scope and "never persist" semantics * record oracle/no-op/partial/human baseline scores where available * keep protected/intermediate scoring behind explicit visibility controls Valid evidence artifact kinds include `screenshot`, `video`, `har`, `browser_trace`, `trajectory`, `critique`, `viewer_metadata`, and `verifier_artifact`. Oracle and calibration evidence is required for leaderboard-grade tasks and recommended for all native tasks with hidden tests, subjective rubrics, or LLM/agent-as-judge scoring. ## Interaction Model An interaction declares one **mode**. The modes form the Interaction axis: | Mode | Meaning | Runtime | | ------------------------ | ----------------------------------------------------------------------------------------------------- | ---------------------------------------------- | | `single-shot` | one agent, one prompt, scored once | executable | | `multi-round` | oracle access / progressive disclosure across rounds (`BaseUser`) | executable | | `simulated-user` | a declarative, model-backed user persona drives turns (not a live wire peer) | partial (linear; one active role per scene) | | `multi-agent-sequential` | roles hand off in one shared workspace | partial (sequential handoff only) | | `arena-concurrent` | an assessor agent and the agent-under-test run as live A2A+MCP peers, scored *during* the interaction | target (runtime-deferred; see Open Primitives) | `arena-concurrent` models a live interaction where the reward is produced *during* the exchange (by an assessor agent) rather than by a post-hoc verifier. It is declared in the schema so such tasks parse without loss; its runtime (an A2A bridge for the agent-under-test plus a concurrently-running assessor) is deferred to a later milestone (see *Open Primitives and Roadmap*). Note ACP (BenchFlow \<-> agent) is not A2A (agent \<-> agent); arena needs the A2A leg. The current executable slice is linear `scenes` that reference `agents.roles`. The first team handoff subset is intentionally narrow: a document-declared user loop may execute explicit multi-role scene turns sequentially when `benchflow.teams..handoff` declares `mode: sequential`, `workspace_visibility: shared`, and `trajectory_visibility: none|metadata`. ```yaml theme={null} agents: roles: planner: agent: claude-agent-acp model: claude-sonnet-4-6 implementer: agent: codex-acp model: gpt-5.5 reasoning_effort: high benchflow: teams: default: handoff: mode: sequential workspace_visibility: shared trajectory_visibility: metadata ``` Richer team semantics such as role membership enforcement, summaries, handoff artifacts, parallel teams, branch routing, and full trajectory sharing are parsed as draft surface but must fail closed until a runtime owns them. Simulated users and nudges should be explicit about runtime type: ```yaml theme={null} user: model: scripted stop_rule: satisfied-or-3-rounds private_facts: hidden_need: reveal only after the solver asks for it benchflow: nudges: mode: simulated-user nudge_budget: 2 ``` This deterministic subset compiles into `RolloutConfig.user` as a `DocumentNudgeUser`: the public prompt runs first, private facts stay out of the package metadata and initial solver prompt, and a fact is revealed only after a targeted clarification question. Bounded model-linear simulated users should stay equally explicit: ```yaml theme={null} user: model: claude-haiku stop_rule: satisfied-or-5-rounds benchflow: nudges: mode: simulated-user branchable: true branch_execution: option-kinds-preserved confirmation_policy: destructive_actions: human ``` If a runtime cannot compile `user` into a concrete loop, it should fail closed or mark the field metadata-only rather than silently ignoring the user. Today the first model-linear slice accepts `claude-*`, `gpt-*`, and `gemini*`-style models for linear single- or multi-scene simulated users through `ModelDocumentNudgeUser`. `confirmation_policy: human` installs a fail-closed ACP permission handler unless the caller supplies an explicit `on_ask_user` handler, and the ACP `ask_user` bridge preserves both option IDs and option kinds so reject/allow choices are explicit branchable evidence. Authors may spell the current executable branch slice as `branch_execution: option-kinds-preserved`; `branch_execution: forked-snapshot` fails closed until the user loop is integrated with the Environment snapshot branch engine. The first sequential shared-workspace team handoff slice records `scene`, `role`, `handoff_from`, and `handoff_to` metadata per user round. `branchable` is still not automatic branch execution; interactive approval UI, parallel teams, handoff artifacts, full trajectory sharing, and branch/message-routing policy remain fail-closed target work. ## Compatibility Compatibility must be explicit. A native package can guarantee export only for the subset the target format supports. ```yaml theme={null} benchflow: compatibility: harbor: export: full emits: config: task.toml prompt: instruction.md oracle: solution/ verifier: tests/ pier: export: degraded losses: - benchflow.teams - benchflow.nudges ``` Target compatibility rules: 1. Native authoring rejects unknown root config keys. 2. Foreign adapter import preserves unknown `task.toml` keys outside native config. The first implementation is `import_task_config_toml()`: strict native validation still fails on unknown keys, while compatibility import returns a validated `TaskConfig` plus `InboundCompatibility.config_extra`. 3. Legacy-to-native migration writes preserved foreign keys under `benchflow.compat.extra`: ```yaml theme={null} benchflow: compat: source: harbor extra_paths: - environment.modal.image - steps[0].runner - verifier.reward_kit.metric extra: environment: modal: image: registry.example.com/task:latest steps: - runner: harbor-step-runner verifier: reward_kit: metric: exact_match ``` 4. Split export rehydrates `benchflow.compat.extra` back into `task.toml` without overwriting supported native keys. The export report records `restored_extension_paths` so compatibility is auditable. 5. `build_harbor_roundtrip_conformance_report()` proves the supported split surface across a split -> `task.md` -> split hop: canonical `TaskConfig`, normalized prompt, and environment/solution/tests file-map hashes. 6. Export emits a degraded-export report when the target format cannot express a native concept. The first implementation is `src/benchflow/task/export.py`, which writes a compatibility split layout plus `compatibility/export-report.json`. 7. Mixed native/legacy files are structurally invalid when task config, prompt, oracle/solution, or verifier/tests aliases drift. 8. Compatibility export should emit `task.toml`, `instruction.md`, `solution/`, and `tests/`. 9. BenchFlow import should prefer `task.md` only when compatibility metadata proves the legacy files are equivalent. 10. Export reports include selected definition, selected verifier/oracle dirs, input/output file hashes, alias collisions, restored foreign extensions, and any lost semantics. 11. `tests/` compatibility is path compatibility, not a separate native standard. For native packages, `verifier/` is authoritative. For split packages, `tests/` remains valid and may contain only `test.sh`. 12. Exporters map the selected native verifier tree to `tests/`; importers may preserve split-layout `tests/` without requiring `verifier.md` or rubric files. If both `verifier/` and `tests/` exist, validation should compare normalized file maps and fail closed on drift. 13. ORS/AgentBeats imports/exports live at the adapter boundary: ORS tool-output rewards become declared reward-event artifacts plus a terminal aggregate, and AgentBeats assessor agents become verifier strategies rather than root task syntax. Round-trip guarantees are semantic, not byte-exact. Config equivalence should be checked through canonical `TaskConfig` dumps; prompt equivalence through normalized prompt text; verifier/oracle equivalence through deterministic SHA-256 file maps over regular files. Comments, TOML/YAML formatting, and blank line trivia are not preserved unless a same-format no-op export asks for that. ## Runtime Capability Matrix Current implementation status: | Feature | Parse | Runtime | Next gate | | | ----------------------------------- | ------: | ---------: | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------- | | `task.md` prompt | yes | yes | keep | | | native `verifier/` | yes | yes | keep | | | native `oracle/` | yes | yes | keep | | | verifier `script` strategy | yes | yes | keep | | | verifier `llm-judge` strategy | yes | yes | keep | | | verifier `reward-kit` strategy | yes | partial | safe relative `reward.py` runner executes; declared criteria parse before launch, emit a runtime manifest, require exact metrics, and compute/verify canonical reward; fuller Reward Kit parity remains target work | | | verifier `agent-judge` strategy | yes | partial | verifier-scoped LLM judge over declared inputs; richer ACP-backed judge agents remain target work | | | verifier `ors-episode` strategy | yes | partial | runtime helper writes ORS tool-output rewards to `trajectory/ors-rewards.jsonl`; declared reward responses/event streams normalize into `reward.json` and `reward-details.json`; fuller OpenReward environment import/export remains target work | | | `agents.roles` | yes | partial | `TaskRuntimeView` carries parsed scenes; explicit sequential shared-workspace handoff can switch roles through the user loop | | | `scenes` | yes | partial | prompt composition compiles; multi-role document-user scenes execute only with explicit turns and supported team handoff | | | `user` / `## user-persona` | yes | partial | `model: scripted` + string `private_facts` compiles to `DocumentNudgeUser`; bounded model-linear users compile to `ModelDocumentNudgeUser`; linear single- and multi-scene user loops execute when every scene is single-role, or when explicit multi-role turns opt into sequential shared-workspace team handoff; `confirmation_policy: human` gates ACP permissions fail-closed without an explicit handler; `branch_execution: option-kinds-preserved` preserves option IDs and kinds; forked branch execution, interactive approval UI, parallel teams, and rich handoff artifacts fail closed | | | `benchflow.teams` | yes | partial | supports exactly one `handoff` with `mode: sequential`, `workspace_visibility: shared`, and \`trajectory\_visibility: none | metadata\`; richer team keys fail closed | | `benchflow:` | raw | no | typed document schema after v0.3 stabilizes | | | imported `steps` | yes | no/partial | fail closed per sandbox until implemented | | | root/step artifacts | yes | no/partial | implement collection or fail closed | | | network allowlist | yes | no/partial | per-sandbox capability check | | | separate verifier env | yes | no/partial | materializer plus verifier runner support | | | Windows / TPU | yes | no | fail closed | | | healthcheck | yes | no/partial | fail closed until sandbox healthcheck support lands | | | workdir | yes | partial | absolute non-root paths are materialized; root/relative paths fail closed | | | `reward.json` precedence | yes | partial | prefer JSON when present and reject both-present mismatches | | | metrics aggregate policy | yes | partial | `mean`, `weighted_mean`, and `weighted_sum`; richer engines remain target work | | | `arena-concurrent` interaction (G4) | no | no | add interaction-mode schema now; A2A bridge for the agent-under-test + concurrently-running assessor at M2 | | | hybrid reward envelope (G1) | partial | no | declared cross-surface product/sum of factors; M1 | | | GAIN aggregation (G2) | no | no | dynamic live baseline + ceiling; M1 | | | leaderboard-submission (G5) | partial | no | hosted / hidden external scorer with durable result record; M1 | | | RL step-reward (G6) | no | no | per-action environment reaction; M1/M2 | | Today `bench tasks check` is structural by default. `bench tasks check --level schema` checks only the task authoring entrypoint and prompt parse, so schema-only fixtures can be validated without pretending to be runnable task packages. `bench tasks check --sandbox ` runs the sandbox-aware capability gate for this matrix, and `bench tasks check --level runtime-capability --sandbox ` names that gate explicitly. `bench tasks check --level publication-grade` adds the first static native publication gate: the package must use `task.md`, native `oracle/`, native `verifier/verifier.md`, rubric files, and selected verifier strategy artifacts with an explicit `reward_json` output contract. Unknown sandbox names fail runtime-capability validation instead of becoming no-op checks. `bench tasks check --level acceptance` adds the first static evidence gate: `benchflow.evidence` must declare oracle proof, verifier reruns and flake rate, a verifier stability report with concrete run records, anti-cheat and instruction-alignment review status, calibration bounds, reference artifacts, a calibration report with no-op/known-bad/partial/reference cases, and SHA-256 pins for every primary evidence file. The static gate parses the declared JSON artifacts and checks that oracle rewards, review status, calibration examples, calibration report cases, and verifier run records agree with the declared metadata. `bench tasks check --level acceptance-live --sandbox ` now adds the first executable live-evidence slice: `benchflow.evidence.acceptance_live.cases` can declare fresh verifier cases and executable `oracle/solve.sh` reruns that pass through the selected sandbox and BenchFlow verifier boundary, with reward thresholds checked from live `reward.txt` / `reward.json` output. `acceptance_live.calibration.from: calibration.report` can also generate live no-op/known-bad/partial/reference cases from the static calibration report; generated low/partial cases require single-line sandbox commands in the report so the checker runs real perturbations instead of treating missing artifacts as expected verifier errors. When `acceptance_live.report` is declared, the runner writes a package-local `acceptance-live-report` JSON artifact plus a `.sha256` sidecar with run records, reward summaries, task/spec hashes, generated/declared case sources, staged workspace hash, and secret-safe diagnostic fields such as `verifier_error_category`, `diagnostic_code`, and `artifact_hint`. Dependency install flakes point to `verifier/test-stdout.txt` instead of embedding raw resolver output in the report. For dogfood against checked-in examples that must not dirty the package, `bench tasks check --level acceptance-live --report-output ` writes the report and sidecar to a host path instead of the package-local path, and `--no-report-write` skips writing the report and sidecar entirely (validation only); the package-local write is reserved for intentionally refreshing checked-in evidence. Repeated live cases can declare `expect.flake_rate_max` to enforce observed flake rate across fresh sandbox reruns; without that field, each failed rerun fails the gate. Larger repeated flake campaigns, model/submission metadata, hosted leaderboard publication, and leaderboard export are still target work. If `acceptance_live.leaderboard.required: true` is declared, the live report includes a local `leaderboard_suitability` verdict requiring live runs, all runs passing, an observed flake rate within `acceptance_live.leaderboard.max_flake_rate`, oracle/reference proof, and generated calibration coverage for no-op, known-bad, partial, and reference cases. A parsed field that the selected runtime cannot honor is worse than a parse error. ## Architecture Slices P0: Add `TaskPackage` / `TaskRuntimeView`. The first runtime-facing slices exist in `src/benchflow/task/runtime_view.py` and `src/benchflow/task/package.py`. They answer: * which entrypoint is authoritative * what prompt goes into `/instruction.md` compatibility materialization * which verifier/oracle directories are native versus legacy * which scenes were parsed for execution * which source hashes and compatibility metadata apply * which verifier document and selected strategy apply * which sandbox runtime issues block launch * which compatibility export report describes target-format loss * which prompt plan composes base, role, scene, and turn prompts for rollout, with redacted document-declared user metadata The remaining package-boundary work is richer adapter import/export state, acceptance/calibration validation levels, and freezing selected verifier/user runtime semantics through launch instead of reparsing at every edge. P1: Add fail-closed capability checks. Rollout, verifier, hardening, and adapters should stop parsing fields they do not execute without surfacing a validation result. This includes explicit gates for `task.md` plus split-file drift, `verifier/` plus `tests/`, and `oracle/` plus `solution/`. The first module is `src/benchflow/task/runtime_capabilities.py` with a pure validator: ```python theme={null} validate_task_runtime_support(task, *, sandbox, task_dir) -> list[UnsupportedTaskFeature] ``` It reports stable config paths and reasons for unsupported `steps`, root/step `artifacts`, allowlists, separate verifier environments, Windows, TPU, healthchecks, unsafe workdirs, document-only `user`/`benchflow` runtime semantics, and non-`main` verifier services on backends that cannot run them. It is wired into `bench tasks check --sandbox ` and the shared sandbox factory used by rollouts and `Environment.from_task()`. Unsupported parsed semantics now raise `UnsupportedTaskFeatureError` before Docker, Daytona, or Modal construction. Safe absolute non-root `environment.workdir` values are materialized before agent and verifier setup. P2: Split native and adapter validation modes. Native authoring should be strict. Foreign import should preserve and warn. P3: Type the `benchflow:` namespace. Start with `document_version`, `compatibility`, `provenance`, `assets`, `secrets`, `evidence`, `teams`, `nudges`, `prompt`, `agent_policy`, and `runtime_policy`. P4: Extend exporters. The first `bench tasks export` path exports to a compatibility split layout with explicit loss reports, backed by `export_task_to_split_layout()`. Extend the same reporting discipline to external benchmark datasets and same-format no-op exports. # Use cases Source: https://docs.benchflow.ai/use-cases # Use cases BenchFlow's Scene-based lifecycle enables evaluation patterns that go far beyond single-turn "prompt and score." This document covers the key use cases for multi-turn, multi-agent, and stateful environment evaluation. The patterns below are all variants of one primitive: **Scenes with Roles and Turns**, all running in a single shared sandbox via ACP. No sidecar containers, no Docker Compose networking — every role lives in the same workspace and talks through ACP. > **Sandbox paths used in the prompts below.** The runtime stages the task > instruction at `/instruction.md` (sandbox root), and the oracle at `/oracle` > for native `task.md` tasks (legacy split-layout tasks use `/solution` as an > alias). The agent workspace is `/app`. For a turn with no explicit prompt > (`Turn("role")` / a bare `- role:` entry), the runtime passes the task goal > **inline** — for native `task.md` tasks it reads the prompt body from > `task.md` and sends it directly, so the agent doesn't have to read > `/instruction.md` to know the task. `/instruction.md` is still staged for > every task, so a role with an explicit prompt can read or quote it. Use > `/oracle` first and fall back to `/solution` if you support both layouts, > e.g. `cat /oracle/solve.sh 2>/dev/null || cat /solution/solve.sh`. *** ## 1. Interactive User Simulation A "user" role provides instructions iteratively; the agent responds. The user has oracle access to the solution and reveals information gradually, simulating realistic human-agent interaction. In BenchFlow, this is a two-role Scene where the "user" role is just another agent with a different prompt and (optionally) a different model. Both roles share one sandbox and one ACP session — no sidecar container, no Docker Compose networking. ### YAML ```yaml theme={null} source: repo: benchflow-ai/skillsbench path: tasks environment: daytona concurrency: 64 scenes: - name: interactive-assist roles: - name: user agent: gemini model: gemini-3.1-flash-lite-preview - name: assistant agent: claude-agent-acp model: claude-sonnet-4-6 turns: - role: user prompt: | You are simulating a user who needs help with the task in /instruction.md. You have access to the oracle solution at /oracle/solve.sh (legacy tasks: /solution/solve.sh). Give the assistant a high-level description of what you want. Do NOT reveal implementation details yet. Write your guidance to /app/user-guidance.md. - role: assistant - role: user prompt: | Read the assistant's work in /app/. Compare against /oracle/solve.sh (legacy: /solution/solve.sh). If incomplete, provide a targeted hint (one specific detail from the solution). Update /app/user-guidance.md with the targeted hint. - role: assistant prompt: "The user provided additional guidance. Read it and continue working." - role: user prompt: | Final check. Read /app/ and compare to /oracle/ (legacy: /solution/). If correct, write LGTM to /app/user-guidance.md. If not, give one final hint. - role: assistant prompt: "Address the user's latest feedback and finalize your solution." ``` ### Python ```python theme={null} from pathlib import Path import benchflow as bf from benchflow.rollout import RolloutConfig, Scene, Role, Turn config = RolloutConfig( task_path=Path("tasks/my-task"), scenes=[ Scene(name="interactive-assist", roles=[ Role("user", "gemini", "gemini-3.1-flash-lite-preview"), Role("assistant", "claude-agent-acp", "claude-sonnet-4-6"), ], turns=[ Turn("user", "You are simulating a user. Read /instruction.md..."), Turn("assistant"), # None = native goal passed inline (legacy: instruction.md) Turn("user", "Check the assistant's work against /oracle/ (legacy: /solution/)..."), Turn("assistant", "The user provided additional guidance..."), ]), ], environment="daytona", ) result = await bf.run(config) ``` ### Why this design * One sandbox, one ACP session — no sidecar container, no Docker Compose networking, no extra server to maintain. * Roles share the sandbox filesystem; any handoff is explicit task state, such as a file named in the next prompt. BenchFlow does not inject messages between turns. * The user agent is a real LLM with full tool access — it can read files, check outputs, and give nuanced feedback, not just templated responses. * Same task folder works for single-turn (baseline) and interactive (with user) via different YAML configs. ### Lighter-weight alternative: `BaseUser` callback When you don't need a second LLM and your "user" logic is rule-based or oracle-guided (e.g. compress instruction → show test failures as hints → stop on pass), use a `BaseUser` Python callback instead of a multi-role Scene. See [progressive-disclosure.md](./progressive-disclosure.md). Built for the SWE-bench Pro progressive-disclosure use case. *** ## 2. Code Review Loop (followup-bench) A coder agent solves the task, then an independent reviewer agent critiques the solution. The coder revises based on the feedback. The reviewer never has write access to `/app/` -- it can only read and provide feedback. ### YAML ```yaml theme={null} source: repo: benchflow-ai/skillsbench path: tasks environment: daytona concurrency: 64 scenes: - name: review-loop roles: - name: coder agent: gemini model: gemini-3.1-flash-lite-preview - name: reviewer agent: gemini model: gemini-3.1-flash-lite-preview turns: - role: coder - role: reviewer prompt: | You are an expert code reviewer. Read the task at /instruction.md and the coder's work in /app/. Write specific, actionable feedback. IMPORTANT: Do NOT modify any files in /app/ except /app/review-feedback.md. Write your specific feedback to /app/review-feedback.md. - role: coder prompt: "Read /app/review-feedback.md and revise your solution." ``` ### Python (with MCP reviewer sidecar) For stronger isolation, use the MCP reviewer server pattern. The reviewer runs as a sidecar service -- it has no filesystem write access at all. The coder calls the reviewer via a tool call: ```python theme={null} from pathlib import Path from benchflow.experimental.mcp.hooks import mcp_reviewer_hook import benchflow as bf from benchflow.rollout import RolloutConfig, Scene, Role, Turn config = RolloutConfig( task_path=Path("tasks/my-task"), scenes=[ Scene(name="solve-and-review", roles=[Role("coder", "gemini", "gemini-3.1-flash-lite-preview")], turns=[ Turn("coder"), Turn("coder", "Call the review_code MCP tool to get feedback, then fix issues."), ]), ], environment="daytona", pre_agent_hooks=[mcp_reviewer_hook(port=8100, model="gemini-3.1-flash-lite")], ) result = await bf.run(config) ``` The MCP reviewer server (`benchflow.experimental.mcp.reviewer_server`) runs as a background process in the sandbox. It exposes `review_code` and `get_review_status` tools via streamable-http. The reviewer LLM reads the code but has **no ability to write files** -- all it can do is return feedback text. ### Results Compare reviewer variants on your task set across three conditions: | Condition | Description | | --------------- | ----------------------------------------------------------- | | `baseline` | Single-agent, single-turn | | `reviewer` | Coder + plain reviewer + coder revision | | `reviewer+spec` | Coder + reviewer that re-reads instruction + coder revision | Treat reviewer lift as an empirical question for the target benchmark. It is most relevant for tasks that require debugging or multi-file coordination, but it should be measured rather than assumed. ### Why this design * No Docker Compose, no sidecar container, no FastMCP server to maintain. * The MCP hook pattern gives the reviewer tool-level isolation: it cannot write to the workspace, preventing reward hacking via reviewer collusion. * Same task, same verifier -- define roles and turns in `RolloutConfig` or rollout YAML. *** ## 3. Skill Generation (BYOS -- Bring Your Own Skill) An agent generates a task-specific skill before solving. This is a two-scene rollout: `prep` (unscored) and `solve` (scored). Both scenes share the sandbox, so the generated skill persists. ### YAML ```yaml theme={null} source: repo: benchflow-ai/skillsbench path: tasks environment: daytona concurrency: 64 scenes: - name: skill-gen roles: - name: gen agent: gemini model: gemini-3.1-flash-lite-preview turns: - role: gen prompt: | Read /instruction.md. Analyze the task requirements. Write a skill document to /app/generated-skill.md that will help an agent solve this task. Include: key steps, common pitfalls, relevant commands or APIs, and a solution outline. - name: solve roles: - name: solver agent: gemini model: gemini-3.1-flash-lite-preview turns: - role: solver ``` ### Python ```python theme={null} from pathlib import Path import benchflow as bf from benchflow.rollout import RolloutConfig, Scene, Role, Turn config = RolloutConfig( task_path=Path("tasks/my-task"), scenes=[ Scene(name="skill-gen", roles=[Role("gen", "gemini", "gemini-3.1-flash-lite-preview")], turns=[Turn("gen", "Analyze the task and write a skill to /app/generated-skill.md")]), Scene(name="solve", roles=[Role("solver", "gemini", "gemini-3.1-flash-lite-preview")], turns=[Turn("solver")]), # None prompt = native goal inline (legacy: instruction.md) ], environment="daytona", ) result = await bf.run(config) ``` ### How scenes work here 1. **Scene 1 (`skill-gen`)**: The `gen` agent reads the task instruction, analyzes it, and writes a skill file. This scene is unscored -- its output is an artifact that persists in the sandbox filesystem. 2. **Scene 2 (`solve`)**: A fresh agent session starts (no context from scene 1). The `solver` agent gets the standard task goal as its prompt (passed inline for native `task.md` tasks; legacy tasks read it from `/instruction.md`) and also sees `/app/generated-skill.md` on disk. The verifier scores only the final `/app/` state. The key insight: `disconnect()` between scenes kills the agent process, so there is no context bleed. The only communication is through the shared filesystem. ### Research findings From the SkillsBench paper: self-generated skills with generic prompts yield approximately 0 percentage points of lift over baseline. The BYOS pattern only helps when the skill-generation prompt is task-type-specific (e.g., "write a skill for compiler tasks" vs. "write a skill for this task"). This result informed the GEPA (Guided Evolution of Prompts and Agents) skill improvement pipeline. *** ## 4. Multi-turn Conversation The same agent receives multiple prompts in sequence, maintaining full conversation context between turns. This is the simplest multi-turn pattern -- no role switching, just sequential prompts to a persistent ACP session. ### YAML ```yaml theme={null} source: repo: benchflow-ai/skillsbench path: tasks environment: daytona concurrency: 64 scenes: - name: iterative-solve roles: - name: solver agent: gemini model: gemini-3.1-flash-lite-preview turns: - role: solver - role: solver prompt: "Review your solution. Run the tests if available. Check for edge cases and fix any issues you find." - role: solver prompt: "Final check: re-read the original instruction and verify your solution addresses every requirement." ``` ### Python ```python theme={null} from pathlib import Path import benchflow as bf from benchflow.rollout import RolloutConfig, Scene, Role, Turn config = RolloutConfig( task_path=Path("tasks/my-task"), scenes=[ Scene(name="iterative-solve", roles=[Role("solver", "gemini", "gemini-3.1-flash-lite-preview")], turns=[ Turn("solver"), # native goal inline (legacy: instruction.md) Turn("solver", "Review your solution. Run tests. Fix issues."), Turn("solver", "Final check: verify every requirement is met."), ]), ], environment="daytona", ) result = await bf.run(config) ``` ### How it works ACP sessions are persistent -- the agent process stays alive across all turns within a scene. The agent retains full conversation history (tool calls, outputs, reasoning) between prompts. Each `Turn` sends a new `prompt()` call on the existing session. No simulated user is required — the "user" in this pattern is the benchmark framework itself, issuing predetermined follow-up prompts. ### Why this is useful * **Self-review**: The second prompt asks the agent to check its own work, catching obvious errors. * **Iterative refinement**: Tasks that require build-test-fix cycles benefit from explicit prompts to test and iterate. * **Decomposition**: Complex tasks can be broken into phases ("first set up the environment", "now implement the feature", "now write tests"). *** ## 5. Cross-model Review Different models fill different roles in the same scene. A cheap model codes, an expensive model reviews. Role-level model configuration makes this trivial. ### YAML ```yaml theme={null} source: repo: benchflow-ai/skillsbench path: tasks environment: daytona concurrency: 32 scenes: - name: cross-model-review roles: - name: coder agent: gemini model: gemini-3.1-flash-lite-preview - name: reviewer agent: claude-agent-acp model: claude-sonnet-4-6 turns: - role: coder - role: reviewer prompt: | You are reviewing code written by a different agent. Read /instruction.md for the task requirements. Examine the coder's work in /app/. Write specific feedback to /app/review-feedback.md - role: coder prompt: "Read /app/review-feedback.md and revise your solution." ``` ### Python ```python theme={null} from pathlib import Path import benchflow as bf from benchflow.rollout import RolloutConfig, Scene, Role, Turn config = RolloutConfig( task_path=Path("tasks/my-task"), scenes=[ Scene(name="cross-model-review", roles=[ Role("coder", "gemini", "gemini-3.1-flash-lite-preview"), Role("reviewer", "claude-agent-acp", "claude-sonnet-4-6"), ], turns=[ Turn("coder"), Turn("reviewer", "Review the coder's work..."), Turn("coder", "Address the reviewer's feedback."), ]), ], environment="daytona", ) result = await bf.run(config) ``` ### Cost-performance tradeoff The cross-model pattern lets you sweep the reviewer axis independently: | Variant | Coder | Reviewer | Question | | --------------- | ------------ | ------------- | --------------------------------------------- | | Self-review | gemini-flash | gemini-flash | Does same-model review help? | | Cross-model | gemini-flash | claude-sonnet | Does a different model catch different bugs? | | Strong reviewer | gemini-flash | claude-opus | Does a stronger reviewer help a weaker coder? | | Weak reviewer | claude-opus | gemini-flash | Does a weaker reviewer hurt a stronger coder? | Each variant is just a different YAML file -- same task folder, same verifier, different role configurations. This enables controlled experiments on the marginal value of reviewer quality. *** ## 6. Stateful Service Tasks Tasks that require agents to interact with live services -- Gmail, Calendar, Docs, Drive, Slack. Services run as sidecar processes in the sandbox, exposing REST APIs on localhost. The agent interacts with real HTTP endpoints, not mocked tool calls. ### Python ```python theme={null} from pathlib import Path import benchflow as bf from benchflow.rollout import RolloutConfig, Scene, Role, Turn from benchflow import SERVICES, build_service_hooks # Declare which services the task needs services = [SERVICES["gmail"], SERVICES["gcal"], SERVICES["slack"]] config = RolloutConfig( task_path=Path("tasks/schedule-meeting-from-email"), scenes=[Scene.single(agent="gemini", model="gemini-3.1-flash-lite-preview")], environment="daytona", pre_agent_hooks=build_service_hooks(services), ) result = await bf.run(config) ``` Service hooks are explicit today. `RolloutConfig.services` is reserved metadata; it does not start services unless you translate it into `pre_agent_hooks`. ### Service registry BenchFlow ships with 5 built-in services (from the SmolClaws project): | Service | CLI binary | Port | Description | | -------- | ------------- | ---- | -------------------------------------- | | `gmail` | `claw-gmail` | 9001 | Mock Gmail REST API (FastAPI + SQLite) | | `slack` | `claw-slack` | 9002 | Mock Slack API | | `gcal` | `claw-gcal` | 9003 | Mock Google Calendar API | | `gdoc` | `claw-gdoc` | 9004 | Mock Google Docs API | | `gdrive` | `claw-gdrive` | 9005 | Mock Google Drive API | Each service: * Runs as a background process in the same container. * Exposes a health endpoint (`/health`) for startup detection. * Uses SQLite for state -- pre-seeded from the task's `environment/` directory. * Is indistinguishable from the real API from the agent's perspective. ### Example task structure ``` tasks/schedule-meeting-from-email/ ├── task.toml ├── instruction.md # "Read the email from Alice, create a calendar event..." ├── environment/ │ ├── Dockerfile # FROM benchflow/claws-base (has all claw-* binaries) │ ├── gmail.db # Pre-seeded: email from Alice with meeting request │ └── gcal.db # Pre-seeded: existing calendar entries ├── solution/ │ └── solve.sh # Oracle: curl commands to Gmail + GCal APIs └── tests/ └── test.sh # Verify: check gcal.db has the new event ```