DEF-28 in Production: Run 10 Finished in 5.95 Hours at 89.6% Fix Rate¶
Wall clock 5.95 hours. 223 rules remediated, 26 escalated, 21 skipped — 270 total terminal dispositions, 89.6% fix rate against the ex-skip pool. The next-best run in project history was R8 at 11.53 hours and 61.8%. DEF-28 didn't just clear the projection window from ADR-0020 — it cleared it by 22 points on fix rate and halved the wall clock.
What changed between R9 and R10¶
Three commits, shipped together, all skill-side except for one harness Protocol addition:
- DEF-28 —
SkillRuntime.worker_context(item)Protocol method. STIG's implementation pre-fetches the XCCDF datastream once at run start (~5 seconds, 1,523 rule descriptions cached) and surfaces the per-rule description as awork_item_contextsection in every apply_fix prompt. The Worker now has the scanner's contract. - DEF-03 — the dream pass computes a follow-aware category credit signal alongside the legacy success-rate signal.
CategoryCredit.confidence_signalprefers the follow-aware signal when DEF-27 data is sufficient. - DEF-27 coefficient tune — the follow-modifier for "tip retrieved but advice ignored" moved from
0.0to0.3, addressing R9's audit-category regression where the original0.0was retiring tangentially-helpful tips.
No harness changes besides the new optional Protocol method. STIG's CVE-style plug-and-play property held — the new pattern is opt-in via worker_context() returning non-None.
The numbers¶
Five runs side-by-side:
| Run | Wall | Remed | Esc | Skip | Fix rate (ex-skip) | What changed |
|---|---|---|---|---|---|---|
| R6 | 19.08h | 153 | 94 | 23 | 61.9% | pre-MTP baseline |
| R7 | 12.20h | 145 | 104 | 21 | 58.2% | MTP cutover, cryptography regression |
| R8 | 11.53h | 157 | 97 | 16 | 61.8% | DEF-26/27 landed; cryptography absorbed |
| R9 | 13.23h | 147 | 103 | 20 | 58.8% | DEF-27 data accumulated; audit regression |
| R10 | 5.95h | 223 | 26 | 21 | 89.6% | DEF-28 + DEF-03 + DEF-27 tune |
R10 vs R8 (the prior best architecturally-clean run):
- +66 remediated rules (157 → 223)
- −71 escalated (97 → 26)
- −5.58 hours wall clock (11.53 → 5.95)
- +27.8 percentage points fix rate
The per-category breakdown is where DEF-28's contribution is unambiguous:
| Category | R8 fix rate | R10 fix rate | Δ |
|---|---|---|---|
| audit | 36% (31/86) | 93% (79/85) | +57pp |
| other | 50% (11/22) | 82% (18/22) | +32pp |
| integrity-monitoring | 29% (2/7) | 57% (4/7) | +28pp |
| user-account | 29% (2/7) | 75% (6/8) | +46pp |
| ssh | 72% (13/18) | 94% (17/18) | +22pp |
| logging | 73% (8/11) | 91% (10/11) | +18pp |
| privileged-access | 50% (2/4) | 100% (4/4) | +50pp |
| package-management | 86% (12/14) | 100% (14/14) | +14pp |
| authentication | 96% | 100% | +4pp |
| kernel | 97% | 100% | +3pp |
| service-config | 100% | 100% | flat |
| filesystem | 100% | 71% | −29pp |
| cryptography | 25% (2/8) | 25% (2/8) | flat |
| banner | 0% | 0% | flat |
Two patterns visible:
-
Categories with chronic-failure scanner-gap rules (audit, integrity-monitoring, user-account, ssh, logging, other) all improved dramatically. These are exactly the categories where journey/38.7's analysis found the Worker was guessing at scanner contracts the XCCDF descriptions specified concretely.
-
Cryptography didn't move at all. Same 2/8 outcome as R7, R8, R9. These rules need a kernel reboot to take effect — the description text in XCCDF won't fix that. DEF-29 territory.
What this means in the STIG remediation landscape¶
Worth being explicit about the framing, because this result lives in a domain where conventional tooling looks very different from what we built.
The standard approach to STIG remediation — Tenable.sc / ACAS, Tripwire, SteelCloud ConfigOS, DISA's SCC tool, openscap-content's Ansible playbooks — is pre-baked remediation scripts. Domain experts author the remediation for each STIG rule against the SCAP content; the tool executes those scripts; outcomes are deterministic because the scripts were tested. Static scripts have a maintenance burden (every new STIG release needs script updates) but the trade-off is well-understood.
What we built is qualitatively different: an LLM-driven harness that reasons about remediation from rule context, rather than executing pre-baked scripts. In that paradigm, the LLM's input data is the load-bearing variable. The question "what does the agent see about the rule it's working on?" is an engineering question with real failure modes.
DEF-28 is the answer to that question for the STIG case: the agent needs the rule's XCCDF description text — the same operative spec that the human authors of static remediation scripts already had open in front of them when they wrote those scripts. Including the description text in the agent's prompt closes the gap.
The R10 result is not "we figured out STIG remediation." Static-script tools have been doing STIG remediation for years. R10 is "an LLM-driven reasoning-based remediation harness can match or exceed the per-rule success rate of static-script tools, generalizing to novel rules without per-rule script authoring." That's a different value proposition, with different trade-offs, in a different paradigm. The 89.6% number is meaningful within that paradigm.
What's still escalating, and why¶
Twenty-six rules escalated in Run 10. The two categories that account for most of them:
Cryptography (6 escalated) — the FIPS / kernel-crypto-state rules. These need a reboot. The Worker correctly identifies what needs to change (update-crypto-policies --set FIPS, kernel parameters via grubby, etc.); the change doesn't take effect on the running system; the OpenSCAP scanner reports FAIL. DEF-29 (the per-family reboot pattern from CVE) addresses this.
Audit (6 escalated) — six rules out of 85. R10 fixed 79 of the 50 chronic-failure audit rules journey/38.7 identified. The remaining 6 are likely rules where the XCCDF description text is genuinely ambiguous (covers multiple acceptable configurations) and the Worker picked one the scanner didn't accept. Worth a per-rule look but not architecturally interesting.
Banner (1 escalated) — the verbatim DoD banner text rule. The XCCDF description has the required text; the Worker is presumably adding extra whitespace or formatting variation. Worth chasing as a one-off.
Filesystem (2 escalated) — slight regression from R8's 100%. Probably noise on a small category, but the per-rule list would tell. The partition_for_* family is most likely; those rules check whether dedicated partitions exist for /tmp, /var, /var/log, etc. and the Worker can't create partitions at runtime.
What this validates architecturally¶
Three claims that needed empirical support before R10 — all now defensible:
-
The harness generalizes beyond binary-evaluator remediation loops. The DEF-28 enrichment mechanism shipped as a Protocol method (
SkillRuntime.worker_context) was originally motivated by STIG's XCCDF descriptions, but the pattern is broader. Any skill whose evaluator has an authoritative spec (CIS benchmarks, Vuls advisory metadata, detection rule schemas, etc.) can implementworker_contextand benefit. The futures/detection-tuning-skill entry exercises exactly this generalization. -
The DEF-27 coefficient tune fixed R9's regression. R8 → R9 lost ground because the
0.0follow-modifier was retiring tangentially-helpful tips. R10's0.3modifier shows no audit-category regression even though the corpus continued accumulating DEF-27 data. The audit fix rate went from 27% (R9) to 93% (R10) — most of that is DEF-28, but the tuning prevented the formula from undercutting the gain. -
The follow-aware category credit (DEF-03) didn't break anything. The dream pass continues to update lesson confidence; the per-category signal now uses follow-aware aggregation when DEF-27 data is sufficient; no obvious side effects in R10's outcome distribution.
Bets, graded¶
From ADR-0020:
| Bet | Predicted | Actual | Grade |
|---|---|---|---|
| Fix rate | 64-68% | 89.6% | Beat upper bound by 22pp |
| Audit category | 50-65% | 93% | Beat upper bound by 28pp |
| Cryptography | 35-50% | 25% | Below floor — confirms reboot-required floor |
| Wall clock | 11-12h | 5.95h | Beat lower bound by 5h |
The cryptography "miss" is correct in the most useful sense: it confirms that the residual ~6 rules are reboot-required, not Worker-reasoning-bound. That's the cleanest possible motivation for DEF-29 (per-family reboot pattern from CVE), which is the natural next architectural change.
What it took to get here¶
Run 10 is the payoff of a four-run experimental arc:
- Run 7 named the regression. journey/38 wrote down "MTP gave us speed, lost us 8 rules, and the cryptography category collapsed in a way we don't yet understand."
- Run 7 post-mortem (journey/38.5) found the misleading-tip pattern in the cryptography corpus — and incidentally surfaced that the Reflector had been naming the scanner-gap pattern across runs.
- Run 8 landed DEF-26 + DEF-27 (journey/38.7) and recovered fix rate to R6's pre-MTP baseline.
- Run 9 tested DEF-27 with accumulated data and regressed. The same analysis pass that named the regression also identified the bigger finding: 78 rules were failing in every run, and the Reflector had been correctly diagnosing why for the entire history.
- The fix wasn't on the loop side (journey/38.8). It was the inputs. The XCCDF descriptions were on the VM since the first scan in early April. The Worker had never read them.
Four runs of architectural iteration, two of them in the wrong direction, to surface that the dominant constraint on the whole system was a piece of input the agent didn't have. The journey/38.8 entry owned the miss explicitly; this entry grades the fix.
The harness was self-improving along the axis of reasoning and blind along the axis of inputs. The Reflector named the gap from R1 onward. The architecture didn't have a verdict for "fix the inputs, not the reasoning." DEF-28 was the manual version of that verdict; DEF-30 is the architectural version, sitting in deferred for if the pattern repeats elsewhere.
What ships next¶
R10's escalation profile points at one concrete next change:
- DEF-29 — STIG adopts CVE's per-family reboot batching pattern for the remaining cryptography rules. ~6-10 rules' worth of lift, smaller than DEF-28 but well-defined. The harness mechanism (
deferrable_failure_modes+resolve_deferred()) already exists; STIG just needs to opt in.
Beyond DEF-29, the long-tail rules become genuinely hard — they're individually unusual, not architecturally bundled. At ~95% fix rate, the system's bottleneck shifts from "architecture" to "per-rule edge cases that need either better spec or human judgment." That's a different category of work, much more skill-specific than architectural.
The detection-tuning futures doc (futures/detection-tuning-skill, futures/detection-tuning-preflight) is the natural next experimental direction — a third skill that exercises DEF-28's worker_context pattern on a structurally different problem shape. If the pattern holds there, the architectural claim "the harness is genuinely skill-agnostic" gains a second empirical data point.
Related¶
journey/38— Run 7, where MTP arrived and the cryptography regression got named.journey/38.5— the cryptography case that motivated DEF-26 / DEF-27.journey/38.7— the R6-R9 cross-run analysis that surfaced the OVAL/scanner-gap finding.journey/38.8— the retrospective entry on missing the descriptions across seven runs.adr/0020— the engineering decision graded above.futures/detection-tuning-skill— the next skill, exercising the sameworker_contextpattern in a different domain.