Skip to content

How We Missed the Descriptions Across Seven Runs — A Retrospective

The XCCDF rule descriptions have been on the target VM since the first OpenSCAP scan in early April. They sit in /usr/share/xml/scap/ssg/content/ssg-rl9-ds.xml, a 27 MB file Ralph has been running oscap xccdf eval against on every iteration. For every single STIG rule, that file contains the authoritative specification of what makes the scanner say PASS — file paths, exact directive syntax, required string values.

The Worker has never read it.

For seven runs, across every architectural iteration the project shipped, the Worker has been guessing the scanner's contract from the rule's title. The rule named accounts_tmout ("Set Interactive Session Timeout") sits next to a description that literally says "typeset -xr" and "in /etc/profile.d/tmout.sh". The Worker spent ten attempts on it in Run 8 — oscillating through /etc/profile, /etc/bashrc, /etc/environment, with various export syntaxes. The Reflector noticed and named the oscillation. The architect-reengagement loop ESCALATEd. The data has the same shape in every run since R1.

This entry is the retrospective on how that happened. The fix is shipped (DEF-28, ADR-0020). The interesting question is the one Ken asked: how did we miss this?

What we built around the gap

Six months of architecture work happened next to the missing input:

  • Reflexion loop — the foundational pattern. The Worker fails, the Reflector explains why, the next attempt incorporates the lesson. Implicit assumption: the Worker has what it needs to succeed if its reasoning improves.
  • Architect re-engagement — when the Reflector plateaus, the Architect re-enters and decides CONTINUE / PIVOT / ESCALATE. Same assumption: the direction is wrong, not the information.
  • Cross-run memory — V1 lessons, then V2 tips with per-retrieval outcome scoring, then DEF-27's per-retrieval causal attribution. Same assumption: we're carrying knowledge from past runs the Worker needs.
  • Ordering constraints — the audit_rules_immutable fix that closed Run 6's cascade. We taught the harness to sequence rules correctly. Same assumption: structural ordering is the bottleneck.
  • Context graphs — DEF-27, ADR-0019, the entire causal-attribution architecture. The assumption that the signal we used to update memory was undersized.

Every one of these bets was correct in its own terms. Every one of them improved the system. None of them addressed the fact that the Worker was running blind on what the OpenSCAP scanner actually checks.

How the gap stayed invisible

The Reflector named the problem in plain English, repeatedly, across runs. Sampling from R8 alone:

"The Worker is correctly loading rules into the kernel using augenrules, but the STIG scanner is performing a static file analysis of /etc/audit/audit.rules." — audit_rules_dac_modification_chown, attempt 10

"Despite verifying that the rule is active in the kernel via auditctl -l, the evaluator continues to fail." — audit_rules_privileged_commands_sudo, attempt 4

"Pattern identified: Oscillation between configuration files (/etc/profile, /etc/bashrc, /etc/environment, /etc/profile.d/) without verifying the evaluator's specific audit logic." — accounts_tmout, attempt 10

The harness had a name for this — scanner_gap_detected, an event emitted whenever N distinct approaches failed evaluator-gap-style without breaking the system. Run 6 had 205 of these events; Run 7 had 391. We tuned the threshold (DEF-17 in deferred.md). We treated it as a metric to watch.

We never asked the next question. If the evaluator's logic is a static file analysis the Worker can't predict, where does its specification live? Three obvious candidate places, each one a single search away from the answer:

  1. The oscap man page describes XCCDF and OVAL formats.
  2. The datastream file is on the VM — ls /usr/share/xml/scap/ssg/content/ returns it.
  3. The Reflector's text mentions "evaluator" 60+ times across runs.

None of these were chased. The mental model never updated from "we need to teach the agent to reason better" to "we need to give the agent the right input."

Five reasons the miss happened, ranked roughly by weight

1. We treated inputs as boilerplate and outputs as the interesting variable. Every iteration improved the loop's behavior given the inputs — the prompt assembly, the memory retrieval, the ordering, the credit assignment. The inputs themselves — what the Worker sees about the rule it's trying to fix — were set early ("rule_id, title, plan") and never revisited. There's no entry in this journal between entry 06 (tool calling) and entry 38 about what's in the Worker's prompt as a domain question.

2. The "memory" abstraction absorbed all the diagnostic energy. Whenever a category of rules failed, the working hypothesis became "we're not learning the right lessons." That hypothesis was self-confirming — V1 lessons → V2 tips → DEF-27 causal attribution — each iteration produced measurable improvements without ever asking whether the agent had the right spec. The audit category's 27-36% chronic floor looked like a memory problem because everything looked like a memory problem.

3. The Reflector's diagnosis didn't have a place to go in the architecture. When the Reflector wrote "the scanner is performing static file analysis," there was no harness mechanism to act on that. The system could only ESCALATE or PIVOT. "Read the scanner's spec" wasn't a verdict the loop knew how to produce. The diagnosis became noise in the log because nothing read it as a structural recommendation.

4. Categories with high success rates anchored our intuition. Kernel rules (97-100% fix rate) and authentication rules (95%+) made the harness feel "mostly working." The audit category's 27% felt like an outlier explained by "audit rules are just hard." When most categories work, the failures look like edge cases, not a missing input class.

5. The audit category specifically is where the gap lives. STIG audit rules are an unusual case: the kernel's auditctl is a runtime API, but OpenSCAP's check is grep-shaped against a static file. The Worker can correctly load the rule via auditctl (and verify it with auditctl -l — which it did, hundreds of times), and the scanner will still say FAIL because the file content doesn't match. This particular static-vs-runtime mismatch is unintuitive — most "configure something" rules check the runtime state. The audit category broke that intuition and we missed it because the other categories' intuition kept working.

The actually-interesting structural lesson

The Reflector was right, repeatedly, in plain text, in events emitted to a JSONL file that we read after every run. The system's self-report told us the answer across every run from R1 to R9. We had:

  • The pattern named (scanner_gap_detected event)
  • The mechanism described (Reflector text: "static file analysis", "specific audit logic")
  • The frequency tracked (391 events in Run 7)
  • The right metric to grade improvements (chronic-failure rate per category)

We had everything except the next move. The architecture had no mechanism for "the Reflector identified a structural input gap; act on it." The verdicts available were CONTINUE / PIVOT / ESCALATE — three actions for "try again with the same inputs." There was no fourth verdict for "the inputs are wrong; fix the inputs."

This connects to DEF-12 in deferred.md — SKIP_UNTIL_DEPS was deferred as a fourth verdict for "this can be fixed but other rules first." We named the gap in the verdict space and walked away from it. The next verdict the system actually needed was something like MISSING_INPUT — "the Worker is repeatedly failing in a way the Reflector identifies as a specification mismatch; the harness should enrich the Worker's prompt with the authoritative spec before retrying."

That's an architectural pattern the project hadn't articulated. The harness can read the Reflector's findings and change what the Worker sees on the next attempt, not just let the Worker try again. DEF-28 is the manual version of this — at run start, we pre-fetch the XCCDF descriptions and include them unconditionally. The richer version would be a harness mechanism that detects "scanner_gap_detected for rule X" and dynamically enriches X's apply_fix prompt with whatever the skill can provide.

That's DEF-30, filed today.

What ships in this bundle

Three changes go in for Run 10. The full engineering record is in ADR-0020; the headlines:

  • DEF-28 — SkillRuntime.worker_context(item) -> dict | None. A new optional Protocol method on SkillRuntime. STIG implements it to return the XCCDF rule description; CVE and future skills can opt in with whatever their evaluator's authoritative spec looks like. Ralph.py adds a work_item_context section to the apply_fix prompt when non-None. Pre-fetch happens once at startup (~5 seconds for 1,523 STIG rules). Plug-and-play preserved by construction.

  • DEF-03 — follow-aware category credit in the dream pass. compute_category_credits now computes a per-category signal weighted by outcome × follow_modifier (the DEF-27 causal attribution data), preferred over the legacy success_rate formula when ≥5 retrievals in the category have DEF-27 scores. Lessons in categories where wins came from followed advice gain confidence; categories where wins were lucky neighbors lose it.

  • DEF-27 coefficient tune. The follow-modifier 0.0 → 0.3 for "followed=false, rule passed" — addresses R9's audit-category regression where the original 0.0 was retiring tangentially-helpful tips. Now followed tips earn 3.3× more credit than ignored ones, but ignored-but-present tips keep some weight.

Bets going into Run 10

From ADR-0020:

  • Fix rate: 64-68% (vs R6/R8's ~62%). DEF-28 contributes most.
  • Audit category: 50-65% (vs R8/R9's 27-36%). The 50 chronic audit failures all share the static-file-check pattern DEF-28 addresses.
  • Cryptography: 35-50% (vs 25-33%). Most need reboot (DEF-29 territory); minor improvement is the floor.
  • Wall clock: ~11-12h. Pre-fetch adds 5s; XCCDF section adds ~300 tokens per Worker prompt.

If those numbers land, DEF-28 produces the largest single fix-rate improvement in the project's history — roughly 2× the V1 → V2 memory work, surfacing more potential lift in one architectural change than all of the prior memory architecture iteration combined.

If they don't land, the Worker's reasoning ceiling is closer than the description-deficit explanation suggested, and the next direction shifts toward DEF-29 (reboot adoption) and Worker model upgrades.

The lesson worth carrying forward

The honest answer to "how did we miss it" is that the architecture was self-improving along one axis (the loop's reasoning) and blind along another (what the Worker sees). The Reflector named the problem in clear English across hundreds of events. The architecture had no place to put that information — no verdict for "fix the inputs, not the reasoning" — so it sat as log noise.

A reflexive system that can act on what its reflector says about itself is qualitatively different from one that can only retry. That's an architectural pattern this project will need beyond STIG. The fix is sometimes not in the loop; it's in what the loop reads from.

The Reflector wrote the answer in plain English from R1 onward. The architecture had no verdict for "fix the inputs," so the answer became log noise.

The data is the project's external memory. The Reflector's text in the JSONL is the project's internal memory of what it noticed but couldn't act on. Both are valuable. Both have to be read.

Live and learn.

  • journey/38.7 — the four-run analysis that surfaced this finding.
  • adr/0020 — the engineering decision record.
  • architecture/02 — the prior context-graph work; DEF-28 sits in the same conceptual family of "the architecture has to act on what it observes about itself."
  • deferred.md DEF-29 (reboot adoption — next), DEF-30 (dynamic prompt enrichment via Reflector findings — speculative).