Skip to content

Bets for Run 11: Under Ten Escalations, or the Theory Has a Hole

R10 finished at 89.6% fix rate, 26 escalations, 5.95 hours wall clock — journey/38.9 wrote that up as the best run in the project's history. The per-rule analysis of those 26 escalations sorted them into four buckets:

  • 8-11 reboot-required (FIPS family + grub2_audit_argument + aide_use_fips_hashes)
  • 11 scanner-gap edge cases (banner, mount_options, audit kernel-state, AIDE config nuances)
  • 4 audit-kernel-state (could be reboot, could be audit-subsystem reload)
  • 3 environmental / policy (installed_OS_is_vendor_supported, etc.)

Two changes shipped tonight to address the first two buckets:

DEF-29 — STIG adopts the per-family reboot pattern from CVE (journey/36, journey/37). The StigEvaluator now returns FailureMode.NEEDS_REBOOT for the curated set of 11 reboot-required rules; the harness defers them; StigSkillRuntime.resolve_deferred batches by family (fips and kernel-cmdline), reboots once per family with snapshot rollback on failure, then per-item re-evaluates.

DEF-28-deeper — OVAL criteria join the XCCDF description in the Worker's work_item_context prompt section. For rules like accounts_tmout, the OVAL criteria say:

ALL of:
  - TMOUT value in /etc/profile <= var_accounts_tmout
  - TMOUT value in /etc/profile.d/*.sh <= var_accounts_tmout
  - At least one config file has TMOUT defined
  - All configured TMOUT values must be >= 1

The description alone said "in /etc/profile, e.g., /etc/profile.d/tmout.sh" — implying the Worker could pick one. The OVAL criteria say both files must satisfy the constraint. R10's Worker spent 10 attempts trying to land this rule because it kept setting TMOUT in one place at a time; the criteria make the scanner's actual contract explicit.

ADR-0021 has the full engineering record.

The bets, ranked by how much I'd defend them

These get graded tomorrow. Each is callable.

1. Fix rate lands in [94%, 96%]. Confidence: high.

R10 was 89.6%. DEF-29 conservatively converts 8 of the 11 reboot-required rules (the 8 that R10's Reflector explicitly named as kernel-reboot-required) to remediated. That's +3.2 percentage points alone. DEF-28-deeper converts a subset of the 11 scanner-gap edge cases — call it 5-7 of them — for another +2 to +3 percentage points. Sum: 94.8% – 96%. The risk is on the high end: if DEF-28-deeper helps more rules than projected (some of the audit-kernel-state edges might convert too), we overshoot. If the reboot batches break the VM, we undershoot.

2. Escalations land in [6, 12]. Confidence: medium-high.

R10 escalated 26. The arithmetic above puts the projected new escalation count at 26 − 8 − 6 = 12 on the conservative end, 26 − 11 − 8 = 7 on the aggressive end. "Under 10" was the user's target; I'd bet at even odds we hit it but I'd take 1.5:1 odds against. Most of the residual will be the 3 environmental/policy rules that no architectural change touches, plus 4-7 edge cases that DEF-28-deeper's criteria don't quite capture.

3. Cryptography category goes from 25% to [60%, 80%]. Confidence: high.

R10 had cryptography at 2/8 = 25%. Six of those eight escalations were reboot-required (the FIPS family). DEF-29 should remediate at least 5 of those 6 after the family reboot. Worst case (one reboot family fails healthcheck) is 60%; expected is 75%. Higher than 80% would require DEF-28-deeper to also fix a non-reboot cryptography rule, which is possible but not central to the bet.

4. Wall clock lands in [6.5h, 7.5h]. Confidence: high.

R10 was 5.95h. Two family reboots at ~60-90 seconds each add 2-3 minutes. The OVAL criteria add ~200 tokens per Worker prompt — ~5% increase in token count, basically invisible against the 27-29 tok/s steady-state MTP-driven generation. The remaining variance is the eight or so DEF-29 deferred items running through resolve_deferred at end-of-run, which adds maybe 10 minutes including the per-family snapshot + reboot + verify cycle.

5. First-attempt success rate stays above 90%. Confidence: high.

R10 had 186 of 198 remediations succeed on attempt 1 (94%). DEF-28-deeper's OVAL criteria make the prompt more specific, not less, so first-attempt success should hold or improve. The risk vector is if the criteria are so verbose for some rules that the Worker over-fits the prompt and loses the natural-language reasoning. Watching for it.

6. At least one DEF-29 family fails. Confidence: medium.

The per-family snapshot rollback exists precisely because reboot batches can break the VM (journey/37 took two iterations to get CVE's reboot pattern right). STIG's version is a clean mirror of CVE's so the risk is lower than CVE's first reboot run was, but a healthcheck failure post-reboot on the FIPS family is a non-zero possibility. If it happens, we'd see the family_batch_complete event with passed: false, reason: family_health_failed for that family, and all FIPS items would be marked failed.

7. The DEF-28-deeper criteria help non-obvious rules too. Confidence: low to medium.

The criteria are richer than the descriptions for many rules, not just the 11 escalated ones. Some R10-passed rules might pass more cleanly (fewer attempts, less Reflector intervention) when the prompt has the criteria. If true, attempt-1 success rate climbs above R10's 94%. Hard to predict the magnitude.

What would invalidate the architecture

If Run 11 lands at sub-94% fix rate or above 15 escalations, the implied finding is one of:

  • DEF-29's curated rule list is too narrow (more rules need reboot than we named)
  • DEF-29's per-family reboot is breaking the VM and rolling back too many gains
  • DEF-28-deeper's OVAL criteria are too verbose or too cryptic for the Worker to parse usefully
  • Some R10-passed rule regresses because the OVAL section pushed something out of the prompt's context budget

All four are testable from the Run 11 JSONL alone. The post-run audit will name the cause.

One thing the journey/38.8 retrospective understated

DEF-28-deeper surfaced something worth naming on its own: the OVAL criteria are not just "the description in machine-checkable form." They are often strictly more specific than the description. The accounts_tmout description says "in /etc/profile or /etc/profile.d/.sh" — disjunctive. The OVAL says "TMOUT in /etc/profile AND TMOUT in /etc/profile.d/.sh" — conjunctive. The natural-language description gave the Worker a partial answer; the OVAL gave the full answer.

journey/38.8 framed the miss as "the descriptions had the answer." That's mostly true. But for a subset of rules, even the descriptions were ambiguous and the scanner's actual semantics were only in the OVAL criteria. The retrospective was right but understated the depth of the input gap.

This matters because it generalizes: when an evaluator has multiple sources of truth (natural-language description + machine-checkable criteria), agentic systems benefit from all of them, not just the most human-readable one. The structural lesson from journey/38.8 holds: the architecture has to be honest about what the agent's authoritative inputs are. DEF-28-deeper extended the practice; DEF-30 (in deferred) is the still-unbuilt architectural version of acting on this dynamically.

The post-run grading rubric

Tomorrow's outcome entry should answer:

  1. Did the fix rate land in [94%, 96%]? If outside, which direction and why?
  2. Were escalations under 10?
  3. Did DEF-29 work as a structural pattern? (Were the family reboots clean? Did per-family rollback fire?)
  4. Did DEF-28-deeper help rules outside the 11 named scanner-gap edges?
  5. Did anything that passed in R10 regress in R11?
  6. What does the residual escalation set look like — is there a clean next architectural change, or is it now per-rule edge cases that need either better spec or human judgment?

If answers 1, 2, 3 are yes, the four-run experimental arc (R8→R9→R10→R11) closes cleanly and STIG is at its architectural ceiling for the current paradigm. Future improvements would be per-rule (rule-specific edge cases) or paradigm-level (richer Worker reasoning, larger model, different skill).

If any of 1-3 are no, the next architectural iteration starts from whatever 38.10's retrospective names as the cause.

  • journey/38.9 — the R10 result that set up these bets.
  • adr/0021 — the engineering record being bet on.
  • journey/38.8 — the retrospective this entry adds a footnote to (OVAL criteria > descriptions for some rules).
  • journey/36, journey/37 — CVE's per-family reboot pattern, mirrored by DEF-29.
  • Next: journey/38.11 after Run 11 lands, grading every prediction in this entry.