A harness that rewrites itself

If you're new here: strata→signal is a small workshop that builds things on its own machines and writes up what it measures. This page measures nothing; it is a reading, and every number and every claim about the code carries a receipt: a file and line in the repository at commit 1cfe254e, a page, table or figure of the paper (arXiv 2610.14224v1), or a file in the OpenCode or Codex source at the version Harness² pins. Where a number was read off a chart by eye, the page says so.

The words this page leans on

• Executor. The agent that does the task, with a frozen model inside: OpenCode or OpenAI's Codex, open-source coding agents that run in a terminal.
• Improver (the paper's "editor"). A second agent session that reads the executor's attempts and rewrites the harness files.
• Rollout. One full run of the executor on one task.
• Token. The unit a model reads and writes in, about three-quarters of a word; an agent re-reads its whole conversation at every turn, so its token count grows far faster than its output.
• Context window. The most tokens a model holds at once, the conversation and its reply together.
• Best-of-N. Run the same agent N times and let a model pick the best output: the plain alternative a new method should beat at equal cost.
• k. The budget knob: at k it runs 4k rollouts and 3k editing stages per task. The default is 1.

What a harness is, and what Harness² changes

A coding agent is a model in a loop that is told its tools, calls one, reads the result, and keeps going until it delivers a file. Everything the loop feeds the model besides the task is the harness, "the system around a model that enables it to act as an agent" (README.md:19-21); the paper's first sentence is that "LLM agents' capabilities depend not only on their underlying models but also on their execution harnesses" (p.1).

Harness² exposes eight editable parts (README.md:70-73; paper Table 5, p.22), under OpenCode each a file in a folder (harness2/substrate/opencode.py:42-88): four always in the agent's context window (systemprompt.md, guardrails.md, LongTermMEMORY.md and tool_descriptions/tool_guidance.md, capped at 24,000 characters together because "every token here competes with the agent's own output budget", opencode.py:50-51), three met along the way (subagents it delegates to, skills a trigger line in the first two names, scripts it runs in its shell), and TypeScript plugins, always registered in its tool loop. The shipped starting harness is stubs that name each slot (paper D.1, p.21). The weights are never touched ("keeping model weights fixed", README.md:25).

How one step works

One step at the defaults, k = 1 and parallel mode (harness2/cli/_common.py:34, :117-119), on one task:

 task ──► base run (stock harness)
              │   transcript + delivered files = the evidence
              ▼
   improver A: "workflow" edit     improver B: "domain" edit
   (tools, procedure, stopping)    (what a practitioner knows)
              │                            │
   candidate A run                candidate B run      ← same task
              └─────────────┬──────────────┘
                            ▼
          improver C: compose (merge, adopt one, or KEEP_BASE;
          cite a moment in a transcript for every edit)
                            ▼
                    final run ──► the deliverable that is graded
                            ▼
          improver D: reflect (helped / hurt / no effect / confounded;
          at most one lesson into this domain's library)

The code asserts "R = 4k rollouts and P = 3k improvement stages" per task, reflection outside that ledger (harness2/core/ledger.py:15-33).

It judges without an answer key. Every session is told "there is no rubric, grading criteria, answer key, reference solution or score anywhere in your context" (harness2/core/prompts.py:38-53; a guard withholds grader verdicts, harness2/core/selection.py:330), the compose session must name "a SPECIFIC MOMENT in a trajectory" for every edit it keeps (prompts.py:913-931), and "KEEP_BASE is a real answer, not a failure" (prompts.py:333-335). It is a model's judgement, not a score: its signals "remain sensitive to execution variability and shared model biases" (p.29, §F), and on JobBench the grader is Claude Opus 4.8 (p.19), the same model that edits (p.21).

The lesson library starts empty ("each sweep starts every domain empty", harness2/core/library.py:15-24, :215-222), and later tasks in a run read what earlier ones adopted; removing the library costs 3.1 and 1.8 points (Table 3, p.10), so a single task, like the README's example, runs without that share.

The improver writes code the executor runs. Scripts run through the shell; plugins are copied into the run's .opencode/plugin/ folder and auto-loaded (third_party/lab_runner/opencode/render.py:431-438). Validation checks only that Python compiles and named files exist (harness2/core/tree.py:254-257); the paper's own failure gallery has "a checker written by the editor" passing a harness that scored lower (Figure 14, p.26).

What the paper reports

Setup (p.7; Appendix D, p.21): three benchmarks of professional work, scored 0 to 100 by hidden tests plus a model's marks (WorkBuddy-Bench) or by a model alone (JobBench, LAB) (p.19-20); executors Gemini 3.5 Flash and GLM-5, each under stock OpenCode and stock Codex; improver, selector and reflector "Claude Opus 4.8 running as an agentic session under OpenCode at high reasoning effort"; every number a mean over three shuffled task orders. One step at k = 1, as the authors state it (Tables 1 and 2, p.8 and p.9):

k = 5 adds 2.9 to 3.9 points on WorkBuddy and 3.0 to 4.2 on JobBench (p.9). With one model as both executor and improver at k = 2, all five models tried improve, GLM-5 by 13.7 points on LAB (Figure 8, p.11). GLM-5, the paper's only open-weights model, has 744 billion parameters, 40 billion active (its model card). No model that fits a 24 GB card is tested anywhere in the paper.

The comparison a careful reader wants. Figure 5 (p.9) plots a "best-of-k base" control: the stock agent run k times, the same selector picking one output (Appendix C, p.20). It spends k rollouts to Harness²'s 4k, the text compares them at the same k, and no table prints it, so we read its bars against the gridlines, by eye and then pixel by pixel. On WorkBuddy's code domain best-of-4 reads about 77.0 against Harness²'s 76.5 at k = 1 (Table 1), the control's highest bar there (best-of-3 about 76.4, best-of-5 about 74.9); on the office domain, about 78.3 against 79.5. With Table 1's run-to-run standard deviations (the typical spread between repeated runs) at 1.3 to 3.6 points and no error bars on the figure, one step is within about a point of best-of-4 at equal rollouts on these two domains and neither is shown better; Harness² also spends three editing sessions where best-of-4 spends one selecting session. (The paper's own note: Harness² at k = 2 beats every plotted baseline at k = 5, p.10, 8 rollouts against 5.) No baseline ships with the release (a search of its code and docs).

What a step costs

The paper reports tokens, not dollars or hours: Figure 6 (p.10) plots gain against "total compute per task", "counting all tokens spent before the final deliverable is submitted", on a log axis (each gridline ten times the last). Read by eye, so rough to about a quarter:

One step costs about six to eight times the base agent's tokens (27 ÷ 4.5 ≈ 6; 2.3 ÷ 0.3 ≈ 8); the reflecting session runs after the deliverable, so it is probably outside those totals. Wall time is reported nowhere ("realized latency also depends on available concurrency and rollout duration", p.10), but the caps bound it: 3,600 seconds a rollout in the paper's runs (p.21; the shipped JobBench runner for OpenCode defaults to 2,400, harness2/benchmarks/jobbench.py:499) and 120 steps and 3,600 seconds an improver session (harness2/core/spec.py:36-44), so at most about eight hours a task at k = 1 if nothing overlaps or is retried (retries are allowed: harness2/core/recursion.py:36-38; jobbench.py:801).

On a home card, an illustration, not a measurement. On one NVIDIA GeForce RTX 3090, Mistral Small 3.2 (24 billion parameters, 4-bit) wrote about 48 tokens a second (47.3 to 49.0 across five rounds in A 3090, in its own words). At that speed a JobBench task's 27 million tokens would take about 156 hours to write. But most of an agent's tokens are re-read, not written; reading runs many times faster (forty times on one workload in Reading is fast, writing is slow), and a local server caches the repeated start of a conversation. The paper does not split read from written tokens, so a task on one card is somewhere between hours and days.

Running it on your own models

The documentation assumes Vertex AI, Google Cloud's model service, and a login with gcloud, its command-line tool (README.md:101-103; docs/SETUP.md:41-52); its first setup line, cp .env.example .env, fails (no such file in any of the three commits; the four settings are in docs/SETUP.md:41-46). But the OpenCode path is not hard-wired: an executor or improver id need only contain a slash, a Google project is demanded only for ids starting google-vertex (harness2/config.py:181-189), and the per-run config files Harness² writes carry only agent steps and permissions (harness2/core/proposer.py:260-262), so a provider block in your own OpenCode config merges in. In shape: a provider block of the @ai-sdk/openai-compatible kind (OpenAI's request format, which most local model servers speak) whose baseURL is your server's /v1 address (vLLM, llama.cpp's server or Ollama, programs that serve a model on your machine), naming your model and declaring its limit; plus HARNESS2_SOLVER_MODEL and HARNESS2_IMPROVER_MODEL set to <provider>/<model> (docs/SETUP.md:41-46). The server must support OpenAI-style tool calls ("function calling"), since that is how the agent acts.

We have not run this. It is the shape the code and docs describe, and these are the caveats, each read in the source, not run:

The Codex executor needs a server with OpenAI's newer Responses API and tool calls (docs/SETUP.md:135-138); check that yours has it. Whether a model small enough for one 24 GB card can drive the improver at all, writing skills, scripts and plugins that pass the validator, the paper does not say.

What it will touch on your machine

The README is candid: "Executor network access is unrestricted, and Codex runs with its sandbox disabled" (README.md:166-171), a sandbox being the fence around what a program may touch. LAB and JobBench rollouts and every improver session run directly on your machine (third_party/lab_runner/opencode/run_opencode.py:7-9); only WorkBuddy rollouts get a container each (paper p.19). At run time Harness² itself, which declares no dependencies (pyproject.toml:20), makes no network call but a five-second connection check to your Codex address (harness2/substrate/codex.py:225-228); setup clones repositories, installs packages and downloads datasets (harness2/config.py:110-115; docs/BENCHMARKS.md:102-103). What it launches is another matter; each row was read in the source at the versions Harness² pins (OpenCode at commit 23fb5e05, Codex at tag rust-v0.144.6), not observed on the wire.

Only the WorkBuddy containers turn anything off (third_party/wb_runner/configs/harnesses/opencode/versions/1.14.18.yaml:14-22). They install OpenCode 1.14.18 from npm, JavaScript's package registry (.../harnesses/opencode/docker/Dockerfile:23), the very release pinned (23fb5e05 is tag v1.14.18; OpenCode packages/opencode/package.json:3), and one flag they set, OPENCODE_TELEMETRY_DISABLED, is not one that release defines (flag/flag.ts), so it does nothing.

How to fence it, as practice, not as something we measured:

1. A virtual machine or container, not your working machine, under a throwaway user with an empty home: no ~/.claude, no ~/.ssh, no saved tokens to read.
2. Set up online once (the pinned clones; Bun, the JavaScript runtime OpenCode runs on; the datasets), then cut the network: nothing outbound from that user's processes except to the model server, your home network included.
3. Set OPENCODE_DISABLE_MODELS_FETCH=1, OPENCODE_DISABLE_AUTOUPDATE=1, OPENCODE_DISABLE_LSP_DOWNLOAD=1 and OPENCODE_DISABLE_CLAUDE_CODE=1. These reach OpenCode only; Harness² rewrites Codex's config.toml at setup (install.py:236-247), so on the Codex path only the network fence, or an edit to that step, stops its metrics and live search.
4. Prove the fence bites: run something that should fail, and watch it fail.

Licences

Harness² is Apache 2.0 (LICENSE; pyproject.toml:10), as are the JobBench runner and grader (third_party/licenses/job-bench-eval.LICENSE) and the pinned Codex CLI (LICENSE at tag rust-v0.144.6, read 2026-10-18); OpenCode and LAB's code are MIT (third_party/licenses/opencode.LICENSE, harvey-labs.LICENSE). The exception is WorkBuddy-Bench's code, runners and patches, under a custom Tencent licence: "WorkBuddy Bench IS NOT INTENDED FOR USE WITHIN THE EUROPEAN UNION ... IN THE EVENT OF ANY CONFLICT, THIS CLAUSE SHALL PREVAIL" (third_party/licenses/workbuddy-bench.LICENSE:3, 13-16; docs/BENCHMARKS.md:80-83). Of the task data, JobBench's is MIT by its dataset card, LAB's ships inside its MIT repository with no separate data licence stated in its README, and WorkBuddy-Bench's carries a custom "tencent" tag (all read 2026-10-18). The paper's five other models are proprietary, used under API terms (p.7, p.11, p.19-21); GLM-5's card gives MIT open weights. None of them runs here; this site's licence ledger covers the models that do.

Should you run it?

For our own setup, no, and we did not. None of our products is an agent loop: each makes one model call, or a short fixed pipeline of them, in seconds to minutes, so Harness², which needs an OpenCode or Codex agent delivering files and gains by running the same task four times, would have no product of ours to improve even if it worked. The agents that help write our pages run on Claude Code, which Harness² uses only as an optional improver forced onto Google's cloud, never as an executor (proposer.py:107-115).

The test we would run, as a design, not a result, if the 24 GB question ever became worth two or three days of our bench machine: one local model of about 27 billion parameters as both executor and improver, a whole copy on each of two NVIDIA GeForce RTX 3090 cards; sixteen JobBench tasks unseen until the rules are written down; three arms (the stock agent, one step, and best-of-4 through a small program of our own), graded by a model of another family not told which arm it reads; a pass only if one step beats best-of-4 by at least 3.0 points on average and the stock agent with 95 per cent confidence. The time bet, at 25 minutes a run and 30 a session with the stock arm free (it is Harness²'s own first run): 16 tasks × (7 runs × 25 + 5 sessions × 30) minutes ≈ 87 hours of card time, two or three days on two cards, far below the worked example's 156 writing hours because a local server caches each conversation's repeated start and a smaller model need not spend Gemini's 27 million tokens, though slow reading of long conversations could double it.

How to check our work

What we did not do: run any part of it, measure any token, second or point, or try the local-provider shape against a live server.

Who ran this, and thanks

A small (human) team asked the question and approved this page for publication. One AI agent read the repository and the paper and produced a reading with file-and-line receipts; a second wrote this page from it, re-opening most cited files and every cited page; a third checked every claim against its source, and the page was corrected from that check. A few receipts rest on the first agent's reading alone: OpenCode's order for merging settings, the LAB README's licence wording, the WorkBuddy dataset card and our own list of products. The prose was drafted by Fable 5.1 and revised by Claude Opus 5.5. Thanks to the authors for releasing the code with its receipts in it, and for writing down that one of their own agents fetched its grading rubric.

Sources

Read 2026-10-18 (UTC) unless dated otherwise.

• Ruiyao Xu, Yanfei Chen, Zhongying CuiZhu, Bhavana Dalvi Mishra, Yifei Ming, Han Yu, Rujun Han, Chen-Yu Lee and Tomas Pfister (2026), "Harness²: Recursive Agent Harnessing for an Open World", arXiv 2610.14224v1. https://arxiv.org/abs/2610.14224
• google-research/harness2 at commit 1cfe254e; anomalyco/opencode at commit 23fb5e05 (tag v1.14.18); openai/codex at tag rust-v0.144.6: the files named above.
• Model and dataset cards on Hugging Face: zai-org/GLM-5, JobBench/job-bench, tencent/workbuddy-bench; the harveyai/harvey-labs README on GitHub.

A reading, not a measurement. Code at one commit, paper at one version, OpenCode and Codex at their pins, all dated above. Nothing ran.

Corrections and later readings will be added below, each dated (UTC), saying in plain words what changed.