RESULTS.md — append-only
Every number the project reports lives here, in order. Never edit a past row; add a new one. Scores are the mean binary task score on the named split (tasks/README.md). Cost is input+output tokens through the Anchor's proxy.
Environment
| item | value |
|---|---|
| host | Debian 13, Xeon E5-1620 v2 (8 threads, no GPU), 31 GiB RAM |
| sandbox image | python:3.12-slim @ sha256:2fe5997d…ce79, pytest 9.1.1 |
| sandbox | rootless Podman 5.4.2, --network=none --memory=1g --cpus=2 |
| model | ornith-1.5:9b (qwen35 arch, Q4_K_M, 8.95 B) on local Ollama 0.33.2 |
| model settings | temperature 0, seed 0, thinking off, context 16384, one request at a time |
Runtime choice (2026-09-09)
Same GGUF, same CPU, measured with llama-bench (llama.cpp build 434ddbb, -march=native) and Ollama's /api/generate timings:
| runtime | threads | prompt eval tok/s | generation tok/s |
|---|---|---|---|
| Ollama 0.33.2 | 4 | 7.4 | 4.07 |
| llama.cpp native | 4 | 7.26 ± 0.03 | 4.07 ± 0.01 |
| llama.cpp native | 8 | 8.00 ± 0.11 | 3.60 ± 0.13 |
No measurable difference: Ollama runs the same AVX kernels (this CPU has no AVX2/FMA). Ollama stays as the runtime; its OpenAI-compatible endpoint supports tool calls and reasoning_effort: "none". Prompt processing is the dominant cost, so the harness keeps message history append-only to reuse the KV cache between steps.
Phase 1 — task suite
(filled in when the suite is frozen)
Phase 2 — Anchor acceptance (2026-09-09)
| criterion | evidence |
|---|---|
| zero LLM calls, no credentials | TestNoModelCalls: no endpoint literals, no provider SDK, net/http only in proxy.go |
| size | 987 non-blank non-comment LOC (TestAnchorSize, ceiling 1000; target 500 not yet met) |
sandbox cannot read anchor/ or holdout/ | TestSandboxIsolation: real container, walks /, finds no anchor, holdout, reference, test_hidden.py; no network; writes outside /work fail |
| rollback reproduces the score | TestRollbackRestoresSnapshotAndScore: cache hit, identical train/holdout, working copy hash equals the snapshot |
| acceptance rule | TestApplyRejections (8 rejection reasons) and TestApplyAccept |
| reconstructible ledger | TestLedgerReconstructsGeneration: parent snapshot + stored diff hashes to the generation's snapshot |
go test -race ./... green on this host with rootless Podman 5.4.2.
Phase 3 — feasibility run (2026-09-09, before the suite was frozen)
One real end-to-end run through the Anchor (sandbox, proxy, Ollama ornith-1.5:9b) on the exemplar task 001_word_frequency:
| score | hidden tests | steps | wall | tokens in/out |
|---|---|---|---|---|
| 1.0 | 19/19 | 4 | 274 s | 6596 / 571 |
The model read the file, ran the visible tests, rewrote the file with a sorted-key fix and re-ran the tests, then stopped. A first attempt with the task's original 300 s timeout was killed mid-step-2; task timeouts are 1800 s from here on. Not a baseline number: one task, train split, cache warm.
Generations
| gen | date | decision | train | holdout | tokens | anchor LOC | note |
|---|
Built 2026-09-09 22:20 UTC · therealagi.ai · a harness that writes itself, under an immutable core.