Run 8 Recovered, Run 9 Regressed — and the OpenSCAP Scanner Gap Has Always Been the Real Story

The plan going into the experimental arc was: Run 8 = signal lands, Run 9 = data accumulation pays off, Run 10 = DEF-03 makes the consumer use it well. The numbers don't read that story.

The four-run arc, by the numbers

Run	Wall	Remed	Esc	Skip	Fix rate (ex-skip)	What changed
R6	19.08h	153	94	23	61.9%	pre-MTP baseline
R7	12.20h	145	104	21	58.2%	MTP cutover, cryptography regression named
R8	11.53h	157	97	16	61.8%	DEF-26 graded outcome + DEF-27 causal attribution landed
R9	13.23h	147	103	20	58.8%	DEF-27 data accumulated; no code changes

R8 was the cleanest fix-rate recovery the project has seen in a single architectural change — back to R6's level after a month of regressions. R9 was supposed to consolidate that gain and didn't. The R8→R9 delta is −10 remediated rules, +6 escalated, +1.7h wall clock. Nine of those ten lost rules were in the audit category.

Where Run 8's recovery came from

R8's wall-clock economics (11.5h, fastest run ever) came from MTP. The 4.4pp fix-rate jump back from R7 is the DEF-26 + DEF-27 work, and the contribution split is visible in the JSONL.

DEF-26 graded outcome (adr/0019): 11 health-failure attempts in R8 produced the −0.2 outcome rows that have been zero across the entire project history. Pre-DEF-26, those failures were stored as outcome_value = 0, indistinguishable from "tip wasn't relevant." With graded scoring active, retrievals that were in the prompt when the Worker broke the system now subtract from utility instead of contributing nothing. The "harmful" column in audit_memory.py's section 5 went from a structural zero to a real signal.

DEF-27 retrieval ranker effect on the cryptography regression: R7's misleading tips (the five update-crypto-policies advisories named in journey/38.5) got DEF-27-scored manually at noon on May 21 before Run 8 launched. R8's retrieval ranker had that data. The rule harden_sshd_macs_openssh_conf_crypto_policy did surface a different top-5 in R8 (tips 9518, 9519, 10139, 10967, 10968 instead of R7's 9423, 10044, 10079, 10086, 10867) — including tip 10139, which advises the explicit-MACs-in-ssh_config approach that R6's lucky-overwrite script accidentally exploited. The ranker shifted. The Worker tried that approach in R8's attempt 1.

The rule still failed. The regression named in journey/38.5 was real, the fix put the right tips in front of the Worker, and the Worker still couldn't satisfy the scanner. The reason — see below — has nothing to do with the misleading-tip problem.

What Run 9 revealed

R9 ran with no code changes from R8. The retrieval ranker now had two runs' worth of DEF-27 data instead of one. The eviction sweep at end of R8 retired 587 tips using the new follow-aware utility formula. Going into R9 the corpus was substantially curated for the first time.

R9 should have looked like R8 only better. It didn't. Audit category dropped from 36% to 27% — back to R7's bottom. The audit losses cluster on rules where R8's retrieval ranker had been surfacing "tangentially related" tips that the new scoring formula classified as tip_followed = false (followed=False, outcome=1.0 → contribution=0 in the new ranker). Many of those tips were retired in the dream-pass eviction sweep. R9 retrieved different tips for the same rules. The Worker took different approaches. The OpenSCAP scanner verdict on those rules didn't change.

The honest read: the follow-modifier 1.0 / 0.0 / 0.5 coefficients in adr/0019 are too punitive. Treating "tip was retrieved during a success but advice wasn't literally followed" as zero contribution conflates two cases the data shows are different — lucky-neighbor (cryptography case) AND tangential-but-load-bearing (audit case). Tuning the 0.0 to something like 0.3 would preserve some signal from tips whose advice the Worker partially incorporated without literal text matching.

That's a one-line follow-up. The bigger finding from R9's regression is structural.

The chronic-failure taxonomy (the actual answer to "is it truly unfixable")

Across R6, R7, R8, and R9 — four runs, four different memory states, four different Worker behaviors — 78 STIG rules failed every single time. They cluster heavily:

Category	Chronic failures (4/4 runs)
audit	50
other	10
ssh	5
integrity-monitoring	4
user-account	3
package-management	2
privileged-access	1
banner	1
logging	1
authentication	1
cryptography	0 (varied between runs)

Three runs' worth of Reflector commentary, sampled across 8 of these rules, names the pattern explicitly. From audit_rules_dac_modification_chown (10 attempts):

Pattern identified: The "Evaluator Gap" loop. The Worker is correctly loading rules into the kernel using augenrules, but the STIG scanner is performing a static file analysis of /etc/audit/audit.rules.

From audit_rules_privileged_commands_sudo (4 attempts):

The team is successfully adding the rule to the configuration files and verifying it via auditctl -l, yet the evaluator continues to fail.

From accounts_tmout (10 attempts):

Pattern identified: Oscillation between configuration files (/etc/profile, /etc/bashrc, /etc/environment, /etc/profile.d/) without verifying the evaluator's specific audit logic.

From aide_verify_acls (5 attempts):

Pattern identified: Repetitive failure of text-based insertion (prepending/appending) of the acls directive to /etc/aide.conf followed by database initialization.

The recurring word is "evaluator". The Reflector has been correctly diagnosing for months that the Worker satisfies the system (kernel loads the rule, auditctl -l confirms it, the setting is functionally active) but fails the scanner — OpenSCAP's static file analysis is looking for an exact string the Worker hasn't produced.

The rule definition for accounts_tmout makes it concrete. Pulled from /usr/share/xml/scap/ssg/content/ssg-rl9-ds.xml on the target VM:

Setting the TMOUT option in /etc/profile ensures that all user sessions will terminate based on inactivity. ... The value of TMOUT should be a positive integer, exported, and read only. The TMOUT setting in a file loaded by /etc/profile, e.g. /etc/profile.d/tmout.sh should read as follows: typeset -xr TMOUT=

Three specifics the OpenSCAP check enforces:

1. The file must be /etc/profile.d/tmout.sh (not /etc/profile, not /etc/bashrc)
2. The variable must be set via typeset -xr (or equivalent readonly), not export
3. TMOUT must be a positive integer

R8's Worker, across 10 attempts on this rule, oscillated through /etc/profile, /etc/bashrc, /etc/environment, and /etc/profile.d/ with various export syntaxes. The Reflector even named it: "various syntaxes (standard export vs. readonly)". But the Worker never converged on the precise form because the Worker has never read the XCCDF rule description. It works off the rule's title and the Architect's plan. The authoritative specification of what makes the rule pass is sitting in a 27 MB XML file on the VM and the Worker doesn't know it exists.

So is it truly unfixable?

Eight rules sampled from the 78 chronic failures, classified by what the Reflector + the actual XCCDF check imply:

Rule	Reflector says	Real ceiling
audit_rules_dac_modification_chown	scanner does static file analysis	Fixable — needs exact /etc/audit/rules.d/*.rules content
audit_rules_privileged_commands_sudo	rule active in kernel but evaluator fails	Fixable — same scanner-gap pattern
banner_etc_issue	evaluator expects verbatim string	Fixable — needs the OS-specific banner from XCCDF
sshd_disable_kerb_auth	sshd configured correctly, scanner says no	Fixable — need to read the OVAL definition
fapolicy_default_deny	configuration correct, evaluator fails	Fixable — same
accounts_tmout	oscillation between files, never readonly	Fixable — XCCDF description gives the exact answer
configure_crypto_policy	FIPS causes health failure, kernel not FIPS-ready	Reboot-required — structural ceiling
aide_verify_acls	text-based insertion failing repeatedly	Fixable — needs exact acls block format

Seven of eight sampled chronic failures are fixable. They share a single cause: the Worker doesn't have access to the OVAL/XCCDF rule check definition. It's guessing the scanner's contract from the rule's title. The descriptions on disk in /usr/share/xml/scap/ssg/content/ssg-rl9-ds.xml literally contain the answer ("typeset -xr", "in /etc/profile.d/tmout.sh") and the Worker has never read them.

The remaining unfixable category — reboot-required rules like the FIPS subpolicy work — is a real structural ceiling for the current STIG skill. CVE solved this for advisories via the per-family reboot batching architecture (journey/36, journey/37). The same pattern would apply to STIG's reboot-required rules. Whether it's worth importing depends on whether a STIG demo can accept reboot pauses in its narrative.

What ships next

The R9 regression makes the next experimental step clear:

1. DEF-03 (per-tip dream-pass credit) — originally scheduled for the "Run 9 work." R9's regression on data-accumulation alone is the strongest motivation possible for the consumer-side change. The follow signal exists; the dream pass still does category-level credit. Per-tip credit would let the Reflector's own "advice was wrong, advice was right" judgment propagate to tip confidence.
2. DEF-27 follow-modifier tuning — the 0.0 for "followed=false, outcome positive" is too punitive on the audit category. The R8→R9 audit losses are concentrated on rules where tangentially-related tips were retired. Move the coefficient to 0.3; rerun.
3. DEF-28 — give the Worker the XCCDF rule description (new). The 27 MB XCCDF datastream on the VM contains the authoritative scanner contract for every rule. Currently the Worker reads the rule's title and guesses; with the description text in the prompt, seven of eight sampled chronic failures become solvable in principle. This is the largest available lift in the whole memory architecture and it has nothing to do with memory — it's about giving the Worker the right input.

DEF-28 is the headline. The whole 78-rule chronic-failure long tail traces to one missing piece of input. Six runs of memory architecture iteration to discover that the answer was sitting on disk on the target VM the whole time.

The Reflector has been correctly naming the scanner-gap pattern for months. The architecture didn't have a way to act on it. The fix isn't a smarter loop — it's the right input.

What R8 actually proved (the part that worked)

This entry leans on the disappointing R9 result, but R8's recovery shouldn't get lost in it:

• DEF-26 graded outcome works as designed: 135 attempt-1 wins at 1.0, 16 attempts-2-3 wins at 0.8, 6 attempts-4+ wins at 0.5, 15 health-failures at −0.2. The harmful column went from structural zero to live signal in one run.
• DEF-27 LLM judge in production works first-try: 4,085 retrievals scored end-of-run, 4,085 successes, zero failures. The cryptography case 4/4 correct judgment translated to a production pipeline that ran for ~19 minutes against the live vLLM and produced clean data.
• Misleading-tip detector activated: audit_memory.py section 4 went from "no data yet" to "13 misleading tips identified" with one dream-pass run. The architectural fix in architecture/02 is empirically validated against real corpus data.
• R8 fix rate recovered to R6's level (61.8% vs 61.9%) — the cryptography regression named in journey/38.5 was effectively absorbed.

The pieces all work. The R9 result says the combination of signal-side fixes alone isn't enough. The next iteration needs both consumer-side credit (DEF-03) and the missing Worker input (DEF-28).

The grades

From the predictions section in journey/38.5:

The cryptography rules will still fail in Run 8 if we leave the tips unchanged. Right — they did.

After the dream pass at end of Run 8, the five crypto tips will accumulate low tip_followed rates ... and they'll get downweighted in Run 9's retrievals. Right — the R7 tips for the regressed rule were partially demoted; R8 retrieved different tips.

Run 9's cryptography category should partially recover ... Mid-50s would be a satisfying result. Wrong direction — R9 crypto was 2/6 (33%), R8 was 2/8 (25%). Slight improvement on rate, no change in absolute wins, the rules involved are mostly the reboot-required kind anyway.

From journey/38.6: "After Run 8 the story has a different shape." That bet was right — the shape did change. Just not the shape I predicted. R9 sharpened what's still missing.

---

Related

• journey/38 — Run 7's MTP wall-clock win + cryptography regression.
• journey/38.5 — the misleading-tip case that motivated DEF-26 + DEF-27.
• journey/38.6 — context-graph framing.
• adr/0019 — the engineering decision graded above.
• deferred.md DEF-03, DEF-27 (follow-modifier tuning), DEF-28 (XCCDF-to-Worker pipeline — the headline next step).