Skip to content

The Journey

First-person field notes of how gemma-forge was built. Chronological, honest, specific. Each entry is scoped to a single moment in the project — a decision, a discovery, a refactor, or a postmortem — and is meant to be readable on its own.

How to read this

  • Chronological: entries are numbered in the order they happened. Decimal numbers (00.5, 06.5, etc.) mark mid-project entries added retroactively to cover moments the original numbering missed.
  • Self-contained: each entry starts with a one-sentence hook and a "why this is its own entry" section that explains what this moment is about.
  • Cross-linked: every entry lists the related entries at the top (via frontmatter) and links to them in the body where relevant.
  • Tagged: every entry has layer, pattern, moment, and optional domain tags so you can find entries by topic in the site search.

Entries, in order

Phase 0 — Starting from scratch

Phase 1 — The inference layer

Phase 2 — The target VM

  • 04. VM Provisioning — OpenTofu + libvirt v0.9.7 + Rocky 9, and an hour of debugging a GRUB hang caused by missing ACPI features.

Phase 3 — The harness

  • 06. Tool Calling — getting Gemma 4 to actually call tools through vLLM and ADK, and realizing our first "loop" was a script pretending to be an agent.
  • 06.5. The Stateful Loop Refactor — replacing ADK's LoopAgent with a Python-driven outer loop and fresh per-turn sessions.
  • 07. The Skills System — pulling STIG-specific logic into a skill manifest so other use cases are a folder-copy away.
  • 07.5. Virsh Console Fallback — the out-of-band recovery path for when SSH+sudo is broken, and the honest documentation of its current bug.

Phase 4 — Iterating on the architecture

Phase 5 — Observability

Phase 6 — The reflexion architecture

Phase 7 — The overnight run and its aftermath

Phase 8 — The second overnight run and v4

  • 18. The Second Overnight Run — 93 rules remediated (78%), the time-waste ratio in the other 26, and three architectural findings for v4.
  • 19. Standing on Whose Shoulders? — research validation of our choices, the literature landscape, and the v4 interface extraction decision.
  • 20. The Interface Extraction — ripping the engine apart mid-flight: five interfaces, a STIG runtime, and 75 tests that still passed.
  • 21. The Task Graph — from flat queue to live DAG: dependency awareness, conflict detection, and a React Flow visualization.
  • 22. Context Graphs and the Memory Question — the research spiral from decision provenance to NIST requirements to "do we even need a database?" — and how the clutch mechanism answered the question.

Phase 9 — The first complete run and cross-run learning

  • 23. The First Complete Run — 270 rules, 13.5 hours, 85 remediated, 157 escalated — and the discovery that the cross-run memory system was storing everything but teaching nothing.
  • 24. Run 2 — Cross-Run Learning — the fix landed: 59 rules flipped from escalated to remediated, the fix rate jumped 35% → 58%, then Run 2 exposed a new cascade and the uncomfortable question of whether memory that was right yesterday can be wrong tomorrow.
  • 25. Run 3 — When the Learning Curve Bends — 60% fix rate, diminishing returns, the environment fidelity problem showing up in real data.

Phase 10 — The memory-architecture pivot

A note on "V1" and "V2": Entries 26-30 use these labels to refer to the cross-run memory architecture, not the harness (which has been on v5 throughout this arc). V1 memory is the category-level dream pass built in entries 26-27; V2 memory is the structured-tip rewrite built in entry 30. Run 4 (entry 28) ended V1; Run 5 tests V2.

Phase 11 — The V2 memory rewrite

  • 29. The Classifier Cheat and the Honesty Check — a near-miss while building V2: a prompt-tuned classifier that almost snuck a thumb-on-scale into the lesson backfill, and why the fix was not a better prompt but no classifier at all.
  • 30. Building V2 — The Memory Rewrite — the seven-commit, one-day rewrite of the cross-run memory layer (V2, replacing V1's dream pass): structured tips, rule-prefix similarity, per-(tip, rule) hit-tracking, history-based eviction. Ships alongside V1 so Run 5 carries both rankings per prompt. Ends with seven graded bets for Run 5.

Phase 12 — Run 5, the deferred registry, and the CVE pivot

  • 31. Run 5: Grading the Bets — four of seven bets won, three lost. V2 is neutral on aggregate (+0.1pp) but better on loop shape (escalated attempts −35%). The immutable cascade moved from position 50 to position 11 and killed the aggregate; the ordering constraint for Run 6 emerges from that diagnosis.
  • 32. Three Tips, a Dead Clutch, and a Registry — Run 5's post-mortem exposed three separate architectural problems the aggregate hid: tips without causal mechanisms mislead the Worker, the clutch is dead code since V5, and neither consolidation pass auto-runs. Seeded docs/deferred.md as the debt registry.
  • 33. The Second Skill: CVE Response — the pivot: every Track A runtime tuning we were about to ship was STIG-specific vocabulary dressed as architecture work. 90 minutes of research collapsed CVE from 1-2 weeks to one day (Vuls exists, ATLANTIS is open source, CVE-Bench doesn't fit our regime). The commercial landscape check confirmed no vendor ships autonomous host-level execution.
  • 34. Run 6: Ordering Works, Runtime Doesn't — fix rate 61.9% (+5.6pp vs Run 5). The ordering constraint completely closed the audit_rules_immutable cascade (position 84/84 vs 11/83). Mechanism field held at 100% across 781 tips. Auto-consolidation retired 356 low-utility tips. Cost: +4.8h wall time (14.3→19.1h). Runtime is now the binding constraint.
  • 35. Building the CVE Skill in a Day — eleven hours from decision to working MVP. Zero edits to the ralph loop; three harness extension points (FailureMode enum, ordering predicate, skill-dir map). Five ATLANTIS patterns adopted, six skipped. MVP smoke 3/3 first-try — validated plumbing, not architectural value. Four predictions for the first full CVE run.
  • 36. Per-Family Reboot Batching: The Architectural Decision Before We Built It — CVE Run 1 closed 29/29 non-reboot advisories first-try; the reboot-verify smoke proved the architecture fires end-to-end but exposed that batch-all loses attribution on failure. The per-package-family rewrite (pre-implementation) — safer than batch-all, faster than per-advisory, the production architecture the whitepaper deserves.
  • 37. Per-Family Reboot Batching Lands: 44 Advisories, Zero Escalations, One Clean Sweep — the design from entry 36 shipped in a single sprint. Four files moved (interfaces.py, ralph.py, the CVE runtime, the Architect prompt). The smoke against a stock Rocky 9 VM remediated 44/44 advisories in 35.5 minutes — 29 non-reboot + 15 reboot-required batched into 2 families (core-userland + kernel) with per-item attribution and safest-first ordering. The ghost-escalation bug and the Architect-SKIPs-reboot-items bug both got fixed and both stayed dead. resolve_deferred phase: 190s for 2 reboots across 15 items. The production story the whitepaper claims is now the story the code executes.
  • 38. MTP Bought 6.9 Hours and Cost 8 Rules — Run 7 finished in 12.20h vs Run 6's 19.08h (1.57× faster) on the same starting state, same memory, same model — the only change was vLLM 0.21.0 + Gemma 4 MTP speculative decoding (google/gemma-4-31B-it-assistant, num_speculative_tokens=2). Per-role median tok/s held 25.9-29.0 across all 12 hours; 73% MTP acceptance aggregate; 2.46 effective tokens per decode step. The wall-clock economics are decisive. The fix rate dropped 8 rules (153 → 145) with cryptography collapsing 83% → 29% on the same starting state — two open hypotheses (drafter randomness, libvirt-snapshot non-determinism), the morning triage in 38.5 found neither.
  • 38.5. The Cryptography Regression Was Misleading Tips, Not Drafter Randomness — Run 7's cryptography collapse traced to five tips retrieved from prior runs, all advising approach X. R6's Worker ignored them and used approach Y (overwriting ssh_config), which won. All five tips marked outcome_value=1 because R6's rule passed. R7's more in-context-obedient Worker followed the tips literally; the rule failed. The corpus had no signal for "tip was retrieved AND its advice was followed" vs "tip was retrieved but ignored" — both look the same in the outcome record. MTP didn't introduce the misleading tips; it exposed them by making the Worker follow advice more obediently. The fix is per-retrieval causal attribution as a first-class column (see 38.6 and architecture/02).
  • 38.6. Context Graphs Are Not a Future Feature — They're What Run 7's Cryptography Failure Demanded — journey/22 filed the context-graph idea as long-term direction six weeks ago. Run 7's cryptography case converted that abstraction into a bill due tomorrow. The fix isn't a fancier graph database; it's two columns on tip_retrievals capturing causation (tip_followed_llm via batched LLM judge in the dream pass, tip_followed_emb via sentence-transformers cosine). The retrieval pipeline was a knowledge graph being asked to do context-graph work; credit-assignment requires causal links the schema didn't carry. The pattern generalizes wherever a retrieval-augmented agentic system grows enough memory for lucky-neighbor outcomes to look indistinguishable from causal contributions. Foundation Capital's trillion-dollar context-graph thesis in concrete form: the missing column was, literally, the entire thesis.
  • 38.7. Run 8 Recovered, Run 9 Regressed — and the OpenSCAP Scanner Gap Has Always Been the Real Story — R8 landed DEF-26 graded outcomes + DEF-27 causal attribution and recovered fix rate to R6's pre-MTP 61.8%. R9 ran with no code changes — the clean test of "does DEF-27 alone scale with accumulated data?" — and regressed to 58.8%. Nine of ten lost rules were in the audit category, suggesting the follow-modifier coefficient 0.0 for "followed=false but rule passed" is too punitive on tangentially-helpful tips. But the bigger finding was hidden in plain sight: 78 STIG rules failed in every single run (R6, R7, R8, R9). The Reflector has been correctly naming the "Scanner Gap" pattern for months — Worker satisfies the system, OpenSCAP fails because its static file analysis expects a specific syntax the Worker is guessing at. Seven of eight sampled chronic failures are fixable IF the Worker had access to the XCCDF rule description text — which sits in a 27 MB XML file on the target VM and has never been added to the Worker's prompt. DEF-28 captures this. The biggest available lift in the whole memory architecture turns out not to be about memory.
  • 38.8. How We Missed the Descriptions Across Seven Runs — A Retrospective — DEF-28 ships today: XCCDF rule descriptions land in the Worker's prompt, 1,523 rules pre-fetched at run start, plug-and-play via a new SkillRuntime.worker_context() Protocol method. DEF-03 (follow-aware category credit) and DEF-27 coefficient tuning (0.0 → 0.3 for "ignored but rule passed") ship in the same bundle. The interesting half of the entry isn't the engineering — it's the honest retrospective on how we walked past the answer across every run from R1 to R9. The Reflector named the scanner-gap pattern in plain English across hundreds of events; the architecture had no verdict for "fix the inputs, not the reasoning," so the diagnosis became log noise. DEF-30 captures the structural lesson: a reflexive system that can act on what its reflector says about itself is qualitatively different from one that can only retry.
  • 38.9. DEF-28 in Production: Run 10 Finished in 5.95 Hours at 89.6% Fix Rate — the payoff entry. Wall clock 5.95h (3.2× faster than R6), 223 remediated, 26 escalated, 89.6% fix rate ex-skip. Audit category went 27→36→27% across R7-R9 to 93% in R10 — the chronic-failure long tail journey/38.7 named, dissolved by one architectural change. Cryptography stayed at 25%, which is the correct result: these rules need a reboot (DEF-29 territory). ADR-0020's projected fix rate (64-68%) was exceeded by 22 points; projected wall clock (11-12h) by 5h. The framing situates the result against conventional STIG remediation tooling (Tenable, Tripwire, SteelCloud — all static-script execution) to clarify what we built is a different paradigm (LLM reasoning-based remediation) where input data is the load-bearing variable. DEF-28 was the answer to that paradigm's central engineering question.
  • 38.10. Bets for Run 11: Under Ten Escalations, or the Theory Has a Hole — prediction entry, landed before Run 11's outcome. Records seven explicit, gradeable bets: fix rate in [94%, 96%], escalations under 10, cryptography from 25% to [60%, 80%], wall clock in [6.5h, 7.5h], first-attempt success rate above 90%, at least one DEF-29 family fails (medium confidence), DEF-28-deeper helps non-obvious rules. Includes the rubric the next entry should grade against — and a footnote refining journey/38.8's retrospective: OVAL criteria are sometimes strictly more specific than the natural-language descriptions, not just "the description in machine-checkable form." Per STYLE.md's "predictions before outcomes" rule.
  • 38.11. Run 11 — Grading the Bets: Three Hits, Three Misses, One That Cost Us the Headline — Run 11 finished at 90.2% fix rate ex-skip (one-tenth of a point above R10), 4.59h wall clock (sharply faster), 95.2% first-attempt success (up from 94%), but 25 escalations (one below R10). The headline number stalled because the FIPS family reboot's SSH wait window — 24×5s — was calibrated for a non-FIPS reboot, and the dracut FIPS module probe needs longer. All 7 deferred cryptography rules came back as family_exception_runtimeerror. Bets 4, 5, 6 hit; bets 1, 2, 3 missed in the same direction for the same single tuning reason; bet 7 weak-hit. Calibration lesson: 38.10's medium-confidence "at least one DEF-29 family fails" bet was the load-bearing dependency for the high-confidence ones, and 38.10 didn't price that contagion. The two-line fix (deadline-based SSH wait with per-family timeouts) ships before Run 12.
  • 38.12. The Wrong Diagnosis and the RSA Key: A 90-Minute Production-Grade Pivot — Run 12 ran with the bumped 600s FIPS SSH wait and failed identically to Run 11. Manual probing showed VM + network + sshd all healthy in FIPS mode; the actual block was FIPS-mode sshd rejecting our Ed25519 user key (server-sig-algs omits ssh-ed25519 in FIPS). 38.11's "calibration" framing was wrong — there was no knob with a correct setting. The production-grade fix was four commands: generate RSA-4096, virt-customize-inject the pubkey, symlink the canonical key path, snapshot the FIPS-enabled state as the new baseline. Zero repo code changes. Smoke scan on the FIPS-baseline pre-passes 6 of 11 FIPS-family rules, leaves a single-item kernel-cmdline reboot family for DEF-29 to exercise cleanly. Predictions for Run 13: 94-96% fix rate, 8-14 escalations, clean DEF-29 verification, 4-5h wall clock. The two lessons: (a) when bumping a constant 2.5× and the failure lands in the same place 2.5× later, the diagnosis is wrong not under-tuned; (b) production-grade fixes are usually much smaller than the production-grade story.
  • 38.13. Run 14 — The Arc Closes at 90.3%: DEF-29 Verified End-to-End, the Ceiling Named — Run 14 finished at 90.3% fix rate ex-skip, 5.03h wall clock, 97.0% first-attempt success (best in the project's history), with skip count dropping from R13's 13 to 5 (architect prompt fix landed). DEF-29's per-family reboot pattern verified end-to-end for the first time across 14 STIG runs: FIPS family up in 28s, healthcheck passed, all four items per-verified — though the four items themselves still failed verification because the Worker wrote --set FIPS when the scanner wanted --set FIPS:STIG. That's a Worker script quality problem, not an architecture problem. Grades 38.12's five bets: 1 hit, 1 partial, 1 narrow-miss, 2 misses. The four-run experimental arc (R8 → R14) closes; the architectural ceiling for the current paradigm is named at ~90% fix rate. The remaining 10pp decomposes into ~3pp environmental, ~3pp scanner-gap residual, ~4pp Worker script quality. Three named paradigm-level moves to break past 90% (DEF-30 dynamic prompt enrichment, skill-bundled remediation library, larger Worker model) — none committed to. The interesting next move is shipping a second skill (detection-tuning, Sigma rules) and seeing whether its architectural lift curve has the same shape.

Phase 14 — Pen-test goes live, fails honestly, learns concretely

  • 41. The Honest Run 1 vs Run 2 — +0pp, and What the Failures Surface — both runs landed 0/5 PASS, 5/5 ESCALATED. Tied at zero — 20 points short of ADR-0023 §4's locked +20pp threshold. Underneath: Run 2 was 11.5% faster on the same target-count, the Reflector emitted 24 net-active tips that named the lab's egress-filter problem in concrete terms, and the architectural gaps that prevented PASSes are now nameable. Four findings worth more than a contrived pass: (1) the lab network's docker bridge blocks outbound TCP, making every reverse_tcp payload structurally unusable — every msf attempt across both runs completed the exploit but couldn't get the callback session; (2) tip retrieval is over-scoped to target identity — Shiro's "prefer file-write over reverse shells" tip didn't transfer to ES, PHP, or ImageMagick despite being true at the lab-environment level; (3) the dream pass's causal-attribution path was built for STIG's repeating-rule shape and silently falls back to STIG outcomes for pentest's random-sample shape (the warning "no work_items found for run_id=…" is the smoking gun); (4) the Worker prompt's "validate recipe_id" instruction isn't strong enough — the Worker invented recipe IDs (3, 10, 1, 14874, 1234) across both runs. The customer-demo story for now: "the agent's memory caught the lab's egress problem in one run and named it concretely; would have prevented it next engagement if tip retrieval were environment-scoped." Phase 5: fix the four gaps, re-run, expect ≥40% PASS on plausible KEV-listed targets.

Phase 13 — The third skill ships, the fourth skill pivots

  • 39. The Third Skill Ships, and the Harness Isn't STIG-Shaped — detection-tuning (Sigma rule tuning against labeled threat-telemetry corpora) shipped functionally across one weekend: five sub-runtimes wired, manifest-driven dispatch picked it up with zero ralph.py edits, DEF-28 worker_context validated in a graded-skill domain, cross-corpus tip transfer measurable (Run 2 attempt 1 hit F1=0.222 vs cold SDS baseline of 0.12 on first opportunity). But the demo headline was 0/5 PASS in both runs. The PASS bar (P>=0.95 AND R>=0.80) was lifted from generic-ML evaluation; real detection engineers don't tune to it. The corpus labels are noisy by construction (file-level, not event-level). Both compounding problems meant no rule could break the F1≈0.7 ceiling regardless of how good the Worker was. Lesson: a skill that proves out the architecture is not automatically a skill that lands as a customer demo. Detection-tuning set aside as customer demo, retained as internal validation of the graded-outcome + corpus-as-input + cross-run transfer claims. PR #1 (feat/detection-tuning-skill). ADR-0022 captures the architecture.
  • 40. Back to the Drawing Board — the Fourth Skill Becomes Autonomous Pen-Testing — 4-hour window after detection-tuning's Run 2 closed 0/5. The construct's four distinctive properties (patient grind + cross-run memory + sovereignty + token-as-input) used explicitly as the selector for skill #4 — and the four-property test collapsed the brainstorm list to one. Three sharp external pushbacks shaped the architecture: (a) "if you plant it, it's theater" → use third-party-built Vulhub targets; (b) "we'd want the latest intel" → dynamic per-engagement corpus refresh, not static curated playbook; (c) "is the corpus wide enough?" → multi-source union (CISA KEV + NIST NVD + Metasploit + ExploitDB + OSV + GHSA + MITRE ATT&CK). Pre-flight (3 hours, captured in futures/pentest-skill-preflight.md) found 75 CVEs at the strict intersection, tool-chain end-to-end tractable, federal-credibility chain iron-clad. ADR-0023 captures seven decisions including iron-clad binary ground truth (the fuzzy-labels trap that defeated detection-tuning is structurally absent here) and cross-run measurement built into the demo via random target rotation. Process lesson: external pushback on the first architectural sketch shapes load-bearing decisions in a way no amount of in-loop iteration replaces. Cost honestly priced at ~2 weeks; awaiting build commitment.
  • 40.5. Phase 1 Lands Clean — The Intel Corpus Pipeline — same-day continuation of 40 after build commitment landed. Phase 1 built and verified the queryable multi-source CVE intel corpus end-to-end: 26,946 unified CVEs / 33,243 exploit recipes / 17 seconds for full refresh (CISA KEV + 10MB MSF metadata + 10MB ExploitDB + sparse Vulhub clone). Dynamic-runtime-lookup architectural claim from ADR-0023 §1 survived first contact with code — query apache + 2.4.49 returns CVE-2021-41773 (KEV, rank=100.0) with three ranked recipes (msf module first, ExploitDB scripts as fallback). Recipe-table separation (multiple recipes per CVE, ordered by reliability) was the right design call. Fingerprints intentionally NOT auto-generated — pushed to Phase 2 vetting to preserve the "Ralph never wastes attempts on non-vulnerable targets" property. Postgres-bootstrap → migrations → ingestion pipeline now well-paved for the third skill in a row (ADR-0016 Phase D per-skill schema-isolation bet paying back several times over). UPSERT-based refresh is idempotent + incremental; can re-run mid-engagement without disrupting in-flight queries. Took ~3 hours real time vs ~3 days estimate — Phase 1 came in under budget. Phase 2 (target pool vetting) starts immediately. Commit 2719a36.
  • 40.6. Phase 2 Ends With a Real Architectural Finding — Phase 2 closed with a 20-target plug_and_play pool (Vulhub + OWASP + Docker Hub), a vetting harness with HTTP-probe fallback for nmap-blind services, and an end-to-end smoke that surfaced a load-bearing design gap. First smoke run: 5/5 pipeline success, 0/5 candidates returned. Cause: CorpusClient.lookup_candidates() matched only against the 13-entry cve_fingerprints table — fine for a 50-CVE curated playbook, impossible against random rotation over a 27k-CVE corpus. Fix: tiered lookup with a 0.5× rank multiplier — precise fingerprint match preferred, but vendor+product fallback against cve_intel directly fills the gap. Second smoke: 3/5 targets returning candidates, and the money shot — target-weblogic-cve-2017-10271 (Oracle WebLogic Server) recovered exactly CVE-2017-10271 blind via nmap-banner-only query. Lesson: dynamic runtime lookup over a 27k-entry corpus is structurally a different problem than a hand-curated playbook of 50 entries. Tiered fallback isn't a hack — it's the mainline; fingerprints are an optimization on top. The 0.5× multiplier is magic-constant for now; Phase 4's cross-run measurement will tell us if it should become a learnable parameter. Two remaining gaps documented for Phase 3: banner-aware corpus lookup (push product-token parsing into the client) and nmap-blind HTTP-probe fallback inside the Worker recon loop.
  • 40.7. Phase 3 — The Skill Stands Up Without an Agent in the Loop — six commits across six sub-phases (foundation, verification, engines, runtimes, prompts, manifest), each green-lit on first or second smoke. The final harness-wiring smoke loaded the skill via load_skill('network-pentest'), instantiated PentestSkillRuntime, all four sub-runtimes satisfied their Protocols, then opened a real meterpreter session against vulhub/php-CVE-2012-1823 in 22 seconds — all without an LLM in the loop. Build estimate was 3 days; came in under 1. Two architectural findings worth recording: (1) banner-based recon can under-resolve the actual vulnerable component (PHP-CGI hidden behind an Apache wrapper) — fix is the pool_intel block in worker_context that surfaces ground-truth recipes from target_pool.intended_cve_ids so the Worker has both views; (2) ExploitDB Python scripts are a mix of Python 2 and Python 3 — the lazy-fetch + engine wired through correctly but a Python-2-syntax script SyntaxErrors in the driver's Python 3. Reflector will learn this; until then the corpus-quality issue stays in the journal. The skill is structurally complete: prompts, engines, verification, manifest, four sub-runtimes, end-to-end smoke. Phase 4 next: Run 1 cold + dream pass + Run 2 warm + the +20pp cross-run threshold measurement. If we hit the threshold with random rotation preventing memorization, the project has shipped its strongest evidence yet for cross-run learning as a real phenomenon.

Phase 15 — The field catches up

  • 44. The Month Loop Engineering Got a Name — June 2026's "loop engineering" surge (Steinberger's seed, Osmani's codification, Cherny's "my job is to write loops," a 6.5M-view post) draws the exact prompt → context → harness → loop lineage this project picked in April, off the same ancestors (Ralph loop, ReAct, Reflexion). Lined up against the movement's six building blocks: three principles the field now preaches we already had run logs for (time-budgeted persistence, deterministic verification, state off the context window), two places they're genuinely ahead (event triggers, worktree parallelism), and two we ship that the discourse hasn't named yet — a memory that grades its own advice (the dream pass, outcome-driven credit assignment) and a measured capability ceiling (12B 1/21 vs 31B 21/21 on sysctl) showing where the loop stops substituting for model size. A positioning entry, not a flag-planting one.