Harness design (Layer 1)

The editable surface. Everything the Evolver may change is a file listed in harness/manifest.toml; everything else in harness/ is immutable plumbing that exists only so the editable parts have something to plug into.

Layout

harness/
  manifest.toml           editable-path list (invariant I3): `editable = [ "...", ]`
  __init__.py             immutable
  __main__.py             immutable  entry: python -m harness /work
  loop.py                 immutable  agent loop: model → tools → model, until done
  model.py                immutable  wire client over MODEL_SOCKET (openai | anthropic)
  trace.py                immutable  JSONL trace writer
  system_prompt.md        EDITABLE   the system prompt
  tools/
    __init__.py           immutable  loads exactly the four tools below
    read_file.py          EDITABLE
    write_file.py         EDITABLE
    run_tests.py          EDITABLE
    list_dir.py           EDITABLE
  middleware/
    __init__.py           immutable  fixed order: retry, truncation, token_budget
    retry.py              EDITABLE
    truncation.py         EDITABLE
    token_budget.py       EDITABLE
  memory/
    __init__.py           immutable
    memory.py             EDITABLE   empty stub in v1
  tests/                  host-side unit tests, never mounted into the sandbox

Nine editable files. The loop, the wire client and the tool registry are immutable so that the number of tools stays four, the model stays frozen, and a candidate edit can only change *what the model is told* and *how tool calls and results are shaped*, never the evaluation protocol.

Contracts

Tool module. Defines SPEC: dict (name, description, parameters JSON schema) and run(args: dict, ctx: ToolContext) -> str. ToolContext carries the task root (/work/task), the module name, and the trace. Paths are resolved relative to the task root and must stay inside it; the tool returns an error string rather than raising. The description text is part of the editable surface on purpose: small models are very sensitive to it.

Middleware module. Defines a class Middleware with any subset of: before_model(state), after_model(state, reply), on_model_error(state, exc) -> bool (True = retry), after_tool(state, call, result) -> str, should_stop(state) -> str | None (reason string stops the loop). state holds the message list, step counter, and cumulative usage. The order is fixed in middleware/__init__.py.

Memory module. recall(task_prompt: str) -> str returns text appended to the first user message (empty in v1); remember(state) -> None is a no-op.

Model client. chat(messages, tools) -> Reply(text, tool_calls, usage, stop_reason). Two wire formats selected by MODEL_API: openai posts to /v1/chat/completions (Ollama, llama.cpp server, vLLM); anthropic posts to /v1/messages. Temperature 0 and a fixed seed where the API allows it; reasoning_effort: "none" in openai mode so thinking models answer directly. MODEL_OPTIONS (opaque JSON from the operator, passed through by the Anchor) is merged into every request body. The client speaks HTTP over the unix socket in MODEL_SOCKET; it has no other network path.

Loop. Messages are append-only. The first user message is the task prompt verbatim plus whatever memory.recall returns. The model's reply either carries tool calls, which are executed in order and appended as tool results, or carries none, which ends the run. Middleware should_stop can end it earlier (step cap, token cap). On exit the loop writes a done event with the reason and the step count. Nothing in the loop knows about scoring.

Append-only matters here: the local model reuses its KV cache only for a byte-identical prefix, and on this CPU prompt processing is the dominant cost, so rewriting earlier messages would make every step pay for the whole conversation again. Truncation middleware therefore trims a tool result once, when it is produced, and never edits history.

Trace events

One JSON object per line in /work/trace.jsonl:

{"ts", "type": "model_call",  "step", "n_messages"}
{"ts", "type": "model_reply", "step", "text", "tool_calls", "usage", "stop_reason"}
{"ts", "type": "tool_call",   "step", "name", "args"}
{"ts", "type": "tool_result", "step", "name", "result", "truncated"}
{"ts", "type": "error",       "step", "message"}
{"ts", "type": "done",        "steps", "reason", "usage"}

Full tool results are kept: the Evolver diagnoses from these, and the sandbox holds nothing secret.

Baseline choices (Phase 3)

System prompt: short, imperative, tells the model it is fixing one bug in one file, to read the file first, run the tests, edit the whole file with write_file, run the tests again, and stop when they pass.
run_tests runs the public suite with a 60 s timeout and returns the tail of pytest's output.
Retry: 3 attempts on transport errors, no retry on 4xx except 429 (budget), which stops the run.
Truncation: tool results over 6000 characters keep the head and tail.
Token budget: stop after 12 steps or 60k cumulative tokens, whichever first.
Memory: stub.

Built 2026-09-09 22:20 UTC · therealagi.ai · a harness that writes itself, under an immutable core.