Run 14 — The Arc Closes at 90.3%: DEF-29 Verified End-to-End, the Ceiling Named¶

journey/38.12 shipped two fixes and made five predictions for Run 14. The fixes were a one-line self._profile = profile in StigSkillRuntime.__init__ (skills/stig-rhel9/runtime.py:360) and a rewrite of the architect prompt's KNOWN SKIP RULES block to stop telling the Architect to skip everything DEF-29 was built to handle. The predictions named a [94%, 96%] band for fix rate, [8, 14] for escalations, a clean DEF-29 kernel-cmdline verify, a [4h, 5h] wall clock, and a >93% first-attempt success rate.

Run 14 landed.

The five bets, graded¶

Bet 1 — Fix rate in [94%, 96%]. MISS, low.¶

90.3% (233/258 ex-skip). R12 was 90.2%; R13 was 86.8%. The improvement over R13 came from exactly where the two fixes targeted: the architect stopped over-skipping (recovering ~8 grub2 rules) and DEF-29's verify step stopped crashing. The miss against the band came from somewhere we didn't predict (more on this in a moment).

Bet 2 — Escalations in [8, 14]. MISS, high.¶

25 escalations: 21 in-loop + 4 deferred-failed. R12 was 24, R13 was 33. Direction right, magnitude wrong. The residual escalation set is qualitatively different from prior runs — see the breakdown below — but its cardinality didn't move much.

Bet 3 — DEF-29 verifies the kernel-cmdline family cleanly. PARTIAL.¶

The kernel-cmdline family didn't fire. grub2_audit_argument ended up in-loop escalated, not deferred, because the Worker's apply script reported APPLIED without triggering the NEEDS_REBOOT path (the rule check after apply got a different evaluator outcome than expected). What did fire was the FIPS family — and for the first time across all 14 runs, it ran end-to-end without exception:

family_batch_start            items=[aide_use_fips_hashes, configure_crypto_policy, enable_fips_mode, fips_custom_stig_sub_policy]
family_reboot_issued
family_ssh_up                 wait_s=28.0       ← FIPS-baseline reboot, normal speed
family_healthcheck_start
family_healthcheck_ok                            ← mission app survived FIPS reboot
family_verify_start
family_item_verified          rule_id=aide_use_fips_hashes              passed=false
family_item_verified          rule_id=configure_crypto_policy           passed=false
family_item_verified          rule_id=enable_fips_mode                  passed=false
family_item_verified          rule_id=fips_custom_stig_sub_policy       passed=false
family_verify_complete        passed=0  total=4
family_batch_complete         passed=true       ← the BATCH succeeded; the items did not

This is the verification chain we built across journey/36, /37, /38.10, /38.11, /38.12 and never actually got to see succeed. Every prior attempt either timed out at family_ssh_wait_tick (R11/R12, wrong-diagnosis era), crashed at family_exception_runtimeerror: reboot_ssh_timeout after the wait window expired (same), or crashed at family_exception_attributeerror: 'StigSkillRuntime' object has no attribute '_profile' immediately after family_verify_start (R13). Run 14 is the first time the structural pattern itself was observed working.

So the bet's letter is partial — wrong family fired — but its spirit hit. DEF-29 worked. The architecture is proven.

Bet 4 — Wall clock in [4h, 5h]. NARROW MISS, high.¶

5.03h. Three minutes over the band. Within the noise of any reasonable measurement. Effectively a hit.

Bet 5 — First-attempt success rate above 93%. HIT, decisively.¶

97.0% (226/233). The best first-attempt success rate in any STIG run across the project. Likely improvement vector: DEF-28-deeper's OVAL criteria continue to land cleaner apply scripts, and the cleaner architect prompt (Fix 2) removed an entire class of "skip a rule that's actually fixable" noise from the work queue.

The new finding: `family_still_failing`¶

The architecturally-interesting fact is why the 4 FIPS-tail items verified as failed. Not because the reboot didn't work — it worked. Not because the healthcheck didn't pass — it passed. Not because the verify step crashed — it didn't. The OpenSCAP rule check, post-reboot, came back pass=false for each of the four. The harness emitted outcome_reason: family_still_failing and escalated them.

Looking at what each rule actually checks against what the Worker actually applied:

Rule	What scanner checks	What Worker did	Why it still failed
`configure_crypto_policy`	`/etc/crypto-policies/config` contains `FIPS:STIG`	Worker ran `update-crypto-policies --set FIPS`	Wrong policy. STIG requires the combined FIPS:STIG sub-policy, not bare FIPS
`enable_fips_mode`	Both `fips=1` kernel cmdline AND FIPS:STIG combined policy active	Worker enabled FIPS via `fips-mode-setup`; didn't set the combined sub-policy	Half the conjunction satisfied
`fips_custom_stig_sub_policy`	A `STIG.pmod` or `STIG-CUSTOM.pmod` file exists in `/etc/crypto-policies/policies/modules/` with valid syntax	Worker tried to create the module via inline heredoc; the script failed with "Unknown policy 'STIG-CUSTOM'"	Worker doesn't know the module file location and format
`aide_use_fips_hashes`	The `NORMAL =` group definition in `/etc/aide.conf` includes `sha512`	Worker's apply script generated invalid AIDE config syntax (this exact rule failed in R12 with a parse error too)	Worker's `sed`-based edit broke the surrounding config

All four are Worker script quality issues. None is an architecture issue. The deferred-reboot-verify mechanism did exactly what it was supposed to do: snapshot the state before reboot, reboot the family, healthcheck the post-reboot system, re-run the per-rule check, and report what the scanner said. The scanner said no, you didn't actually fix this — and that's an honest report.

This is what an architecture being correct looks like. It doesn't lie about what it can't fix.

The architectural ceiling, named¶

R10 → R14 fix rates: 89.6%, 90.2%, 90.2%, 86.8%, 90.3%. Five runs all clustered in a ~3 percentage point band, despite each one shipping a substantive architectural addition (DEF-28, DEF-29, DEF-29 calibration, DEF-29 + prompt fix). The shape isn't subtle: the current paradigm has its ceiling at roughly 90%.

The 10 percentage points the harness doesn't get are now structurally identifiable:

~3pp environmental / scope-bounded. 5 partition_for_* rules (live-system repartitioning is out of scope by design), plus installed_OS_is_vendor_supported and a handful of policy rules that require humans (fapolicy_default_deny, configure_opensc_card_drivers, postfix_client_configure_mail_alias).
~3pp scanner-gap edges DEF-28-deeper doesn't fully close. A few audit rules (audit_rules_privileged_commands_*, auditd_data_retention_admin_space_left_percentage), the AIDE edges (aide_build_database, aide_check_audit_tools), banner_etc_issue, and the accounts_umask_* family — rules where even the OVAL criteria don't quite capture what the scanner ultimately verifies.
~4pp Worker script quality issues. The 4 FIPS-tail items above, plus 2-3 rules where the Worker generates almost-correct scripts that the scanner rejects on a syntactic detail.

The first bucket is irreducible from inside this paradigm — fixing it would require expanding scope to include build-time decisions (partition layout) or human-in-the-loop signoff. The second bucket is the unsolved-without-paradigm-shift residual that journey/38.7 named as the chronic-failure long tail. The third bucket is the one that's cheaply fixable with paradigm-level moves we haven't yet committed to.

What would push past 90%¶

Three named paradigm-level lifts, in increasing order of investment:

1. DEF-30 — Dynamic prompt enrichment via the Reflector. When the Reflector says "the scanner expects update-crypto-policies --set FIPS:STIG not --set FIPS" — and the Reflector does say this; the Reflector reads the scan output and identifies the gap — the next Worker prompt for that rule includes that specific correction. Currently the architecture has no path for the Reflector's diagnosis becoming the Worker's input. journey/38.8's footnote called this out as the deepest unrealized lesson: a reflexive system is qualitatively different from one that can only retry. DEF-30 is the implementation.

2. A skill-bundled remediation library. A per-rule library of known-good remediation scripts (curated, version-controlled, scanner-verified). For rules where the script is well-known and stable (update-crypto-policies --set FIPS:STIG, the canonical STIG.pmod content, the right sed command for AIDE's NORMAL group), the harness consults the library before the Worker writes a fresh script. This collapses the third bucket essentially completely, at the cost of maintaining a library. Not philosophically clean — it's hand-curated knowledge — but it would be a real STIG remediation tool's right answer, because real STIG remediation tools ship with hardening scripts.

3. Richer Worker reasoning. Larger model, more inference budget per rule, multi-step planning. Generally improves quality, but unpredictably; could just as easily over-think a simple rule into a worse script.

journey/38.12 mentioned each of these in passing. None has been committed to. None is on the path to closing the R8→R14 arc — that's done. They're the path to re-opening the question of what STIG remediation can look like.

What this entry's lesson is¶

The R8 → R14 arc closes with two facts that don't usually coexist:

Every architectural addition shipped did exactly what it was designed to do. DEF-26 (graded outcomes), DEF-27 (causal attribution), DEF-28 (XCCDF descriptions), DEF-28-deeper (OVAL criteria), DEF-29 (per-family reboot batching), DEF-29-calibration (deadline-based wait), the two Fix 1 / Fix 2 corrections — all of them landed and all of them verified.
The aggregate fix rate moved less than the per-DEF rhetoric implied it would. DEF-28 shipped with a projected band of [64%, 68%] and overshot to 89.6% — that's the run that did the heavy lifting (R10). Everything after was rearranging the residual: closing a sub-bucket here, exposing a latent bug there. R14 is 0.7pp above R10.

Both facts are true. The lesson is that a paradigm's ceiling is not the same as the sum of its architectural lifts. Each lift made the architecture more correct at what it claims to do. Together they convinced the workload it was being maximally well-served. But the workload has a ceiling that doesn't move just because the architecture is doing its job better. Past a point, you're not making the architecture better — you're paying the architecture's interest, and the principal stays where it is.

The interesting move from here is not to keep refining STIG. The interesting move is to ship a second skill and see what happens to that skill's architectural lift curve. If detection-tuning (Sigma rules, a different domain entirely) goes through a similar arc — first big-step gain from an input lift, then plateau — that's evidence the pattern generalizes. If detection-tuning's arc looks structurally different, the architecture we have is more domain-specific than we currently believe.

That's what we're going to find out next.

Predictions retired¶

This is the last STIG-prediction entry. The remaining open question for STIG was "is the architecture's design actually correct?" journey/38.13 answers that: yes, verified end-to-end, paradigm ceiling named at 90%. Further STIG work would be either (a) DEF-30 / library work that would shift the paradigm, or (b) per-rule Worker prompt tuning that would feel a lot like polishing.

The next prediction-and-grading arc kicks off when detection-tuning has enough infrastructure to run its first end-to-end attempt. That's the new beat the journal follows.

journey/38.12 — the entry whose predictions this grades; the diagnostic chain that finally got DEF-29 to a working test.
journey/38.11 — the earlier (wrong) grading entry whose "calibration error" framing this entry retires.
journey/38.9 — R10, the paradigm-establishing run that DEF-29 + DEF-29-calibration + the two fixes never substantially exceeded.
journey/38.8 — the entry that named DEF-30 (dynamic prompt enrichment) as the deepest unrealized lesson; this entry promotes DEF-30 from "deferred" to "named next-iteration vector."
adr/0021 — DEF-29's engineering record. Gains a final amendment confirming the architecture verified end-to-end on R14, plus the architectural-ceiling discussion.
Next: the journal pivots to detection-tuning. The first end-to-end attempt is the new beat.