Skip to content

Run 11 — Grading the Bets: Three Hits, Three Misses, One That Cost Us the Headline

journey/38.10 shipped seven explicit, gradeable bets the night Run 11 launched. This is the grading entry. The format is the rubric from 38.10, in order, with verdict and the numeric reason.

The run, in five lines

  • 270 rules in scope (failing at scan).
  • 230 remediated, 18 escalated in-loop, 15 skipped, 7 deferred → all 7 failed at family reboot.
  • Fix rate ex-skip: 90.2% (230 / 255).
  • Wall clock: 4.59h (02:35 → 07:11 UTC).
  • First-attempt success: 219 / 230 = 95.2%.

The bets, graded

Bet 1 — Fix rate in [94%, 96%]. MISS, low.

Landed 90.2%. R10 was 89.6%, so we picked up 0.6 percentage points. The bet projected +5 to +6 from two architectural changes (DEF-29 worth ~3.2pp, DEF-28-deeper worth ~2-3pp). DEF-28-deeper appears to have delivered most of its share — the in-loop escalation count fell from R10's 26 to R11's 18, an improvement that lines up with the OVAL criteria converting some scanner-gap edges. DEF-29 delivered zero of its share, because all seven deferred FIPS items came back as family_exception_runtimeerror. If those seven had verified post-reboot as the bet assumed, the fix rate would have been (230+7) / 255 = 92.9%, still below the band but inside the margin. The bet's underlying arithmetic was roughly right; one tuning constant ate the result.

Bet 2 — Escalations in [6, 12]. MISS, high.

25 escalations: 18 in-loop + 7 deferred-failed. R10 was 26. We picked up exactly one rule. Same diagnosis as Bet 1 — if the FIPS family had reboot-verified cleanly, escalations would have landed at 18, a 31% reduction over R10 and inside the upper edge of the projected band. The bet was directionally right and quantitatively wrong because DEF-29's seven would-be wins flipped to seven losses.

Bet 3 — Cryptography category in [60%, 80%]. MISS, severe.

R10 cryptography: 2/8 remediated = 25%. R11 cryptography (counting deferred-failures as the failure they functionally are): 2/(2+7) = 22%. Cryptography category fell, not rose, because the FIPS family reboot — the entire mechanism the bet hinged on — never got past the SSH wait. The non-FIPS crypto rules (2 of them) remediated cleanly; the four cryptography rules in skip were declined for the usual reasons (BIOS/UEFI scope).

This is the bet that cost us the headline number.

Bet 4 — Wall clock in [6.5h, 7.5h]. BEAT, decisively.

4.59h. The bet projected a slowdown over R10's 5.95h on account of family reboots and prompt-length overhead from OVAL criteria; both showed up but neither dominated. The two family-reboot SSH wait cycles cost ~244s before timing out (one cycle, since only the FIPS family existed in this run). The OVAL prompt addendum stayed within tokens-budget noise. The aggregate effect was that R11 finished substantially faster than R10 because in-loop attempt counts fell — first-attempt success climbed to 95.2% (R10: 94%), and DEF-28-deeper's clearer criteria meant fewer Reflector cycles. The bet was conservative in the wrong direction. Worth noting in the next entry's projection.

Bet 5 — First-attempt success rate stays above 90%. HIT.

219/230 = 95.2%. Up from R10's 94%. The fear was that overly verbose OVAL criteria might confuse the Worker; the data says the opposite — the criteria helped first-attempt success, by closing the description's ambiguities. Of the 11 remediated rules that took >1 attempt, only one took 10 (the same kind of long-tail outlier R10 had), one took 6, two took 4. The bulk of the cost of failure stayed in the escalation bucket, not in attempt churn.

Bet 6 — At least one DEF-29 family fails. HIT.

This was the medium-confidence bet. The FIPS family failed at the SSH wait step, exactly the failure mode 38.10 named: "a healthcheck failure post-reboot on the FIPS family is a non-zero possibility." The actual failure was slightly more boring than that — we never got to the healthcheck because SSH didn't return within the 24-attempt poll window (about 240 wall-clock seconds). The architecture's snapshot-rollback path didn't fire because the harness aborted earlier than the rollback hook expected. Functionally the result is the same: all seven items in the family flipped to family_exception_runtimeerror.

The relevant emit event:

family_ssh_wait_tick { family: "fips", elapsed_s: 243.9, attempt: 24, max_attempts: 24 }
family_ssh_timeout   { family: "fips" }
family_exception     { family: "fips", exception_type: "RuntimeError", detail: "reboot_ssh_timeout" }

The constant was wrong. FIPS-mode kernel boot on this VM is genuinely slower than non-FIPS — the dracut FIPS module probe adds a non-trivial cold-start cost. 240 seconds is enough for a normal Rocky reboot; it is not enough for the first FIPS reboot from a non-FIPS baseline.

Bet 7 — DEF-28-deeper criteria help non-obvious rules. WEAK HIT.

Hard to grade cleanly from one run, but the signals lean weakly positive. First-attempt success climbed; total in-loop escalations fell from 26 to 18 (a 31% reduction); the audit category held at ~96% (R10's number) which is the right outcome — there was less ground to gain there. The remaining 18 in-loop escalations are mostly the environmental and policy-class rules 38.10 named (installed_OS_is_vendor_supported, fapolicy_default_deny, accounts_authorized_local_users) plus a handful of remaining scanner-gap edges (mount_option_boot_nosuid, aide_check_audit_tools) that even the OVAL criteria don't fully capture.

The next iteration would need either DEF-30 (dynamic prompt enrichment via the Reflector) or a per-rule remediation library to push past these.

Aggregate verdict

Three hits (4, 5, 6), three misses (1, 2, 3), one weak hit (7). Five of the six wrong-direction misses trace to a single tuning constant — the FIPS SSH wait window. The architecture's behavior under the failure mode was correct: rules were deferred, the family was batched, the snapshot existed, the timeout was named with a specific event, and the harness continued to a clean end_run instead of crashing the loop. (It crashed elsewhere on the JSON serialization path, which got fixed mid-run-grading and committed as 77eb075.)

What journey/38.10 got right and wrong

Got right: the per-family reboot pattern is the correct architectural response to reboot-required rules; DEF-28-deeper's OVAL criteria help; wall clock direction matched intent (was going to slow down, did not slow down much because attempt-count fell); the medium-confidence prediction of a family failure was correct.

Got wrong: the bet on the magnitude of DEF-29's gain assumed the reboot would succeed. 38.10 named the failure mode but priced it at "non-zero possibility" rather than "if your SSH wait is calibrated to a non-FIPS reboot, this will fail with certainty on the first FIPS family." That's not a forecasting error in the strict sense — it's a calibration error that 38.10 had no way to detect without running the FIPS reboot end-to-end at least once. Which is what this run did.

The journey/38.10 framing — "under ten escalations, or the theory has a hole" — gets a qualified answer: the theory's shape is right, the theory has a hole the size of one named tuning constant, and the four-run experimental arc cannot close at the architectural ceiling until that constant is fixed and re-run.

What ships next

Two-line fix, then Run 12.

The change: deadline-based SSH wait in _reboot_and_verify_family, with a per-family timeout map. FIPS gets 10 minutes; everyone else gets 3. The current range(24) loop with 5s sleeps is replaced by a while time.time() < deadline loop that prints elapsed wall-clock to the same family_ssh_wait_tick event so dashboards keep working. ADR-0021 gets a one-paragraph note added about the calibration delta.

Run 12's prediction: same 90.2% baseline, plus 5-7 of the 7 deferred FIPS items remediating, landing fix rate at 93.0% – 92.9% ex-skip (range narrowed by whether one of the seven FIPS rules has its own non-reboot issue we haven't seen yet). Escalations land at 18-20. Wall clock adds ~6-8 minutes for the longer FIPS reboot itself plus the actual healthcheck and per-item verification. Cryptography category goes from 22% to 78-100% — the bet that 38.10 was originally making.

If Run 12 lands at 93%+ with ≤20 escalations, the four-run experimental arc closes — DEF-29's engineering is right, DEF-29's tuning is right, and STIG hits the architectural ceiling for the current paradigm. The next step from there is either DEF-30 (dynamic prompt enrichment) or moving to the next skill (detection-tuning, on its own branch).

A note on calibration

The pattern across journey/38.10 → 38.11 is worth naming: a prediction entry that names six confident bets and one medium-confidence one turned out to have the medium-confidence bet be the load-bearing one for whether the others would land. Five of the six high-confidence bets were conditionally correct given the seventh held. Calibration is harder than it looks when bets aren't independent.

Future prediction entries should look at the dependency graph between bets and flag the load-bearing ones explicitly. "Bet 6 is the precondition for bets 1, 2, 3 — if it fires, those miss low" would have been an honest framing. 38.10 came close to saying it (the FIPS reboot failure mode was named) but didn't quite price its contagion.

  • journey/38.10 — the prediction entry this grades.
  • journey/38.9 — R10 (the result R11 was built on top of).
  • adr/0021 — the engineering record. Will be amended with the FIPS calibration note.
  • Next: journey/38.12 after Run 12 lands. The four-run arc closes there if the calibration fix delivers what the architecture promised.