The Wrong Diagnosis and the RSA Key: A 90-Minute Production-Grade Pivot

journey/38.11 graded Run 11's seven bets honestly: three hits, three misses, one weak-hit. The three misses all traced to a single constant — the FIPS family's SSH wait window — and 38.11 said "the architecture's shape is right, the theory has a hole the size of one named tuning constant." So we shipped the deadline-based fix (commit c4ebc55), bumped the FIPS window to 600s, and launched Run 12.

Run 12 finished with the same numbers as Run 11. 90.2% fix rate, 24 escalations, 5.25h wall clock. The new 612-second SSH-wait window timed out exactly where the 244-second window had. Then snapshot rollback fired and the kernel-cmdline family timed out too — at the original 180s window, on what should have been a clean baseline.

The first reaction to identical failures across two reboots from supposedly-different starting states should always be the same: the diagnosis from yesterday is wrong, not understated.

What 38.11 got wrong

The journey/38.11 entry's framing — "calibration error... not a forecasting error in the strict sense" — was itself the next forecasting error. The verb "calibration" implies a knob exists with a correct setting. There was no knob. Adding more wait time didn't help because the thing we were waiting for wasn't going to happen.

What 38.11 should have said: "the architecture deferred, batched, snapshotted, rebooted, and timed out — but we don't know what the VM did, because the only signal we have is 'SSH didn't return.' Before adjusting the wait constant, run one direct probe."

That probe — a manual ssh -vv against the broken-state VM — would have taken 90 seconds and shown what we eventually saw: TCP/22 reachable, key exchange clean, "Authentications that can continue: publickey, gssapi-keyex, gssapi-with-mic", server signed-algs list missing ssh-ed25519, and a final "Permission denied (publickey)". Not a network failure. An auth failure. Specifically: FIPS-mode sshd on Rocky 9.7 doesn't accept Ed25519 user keys, and our SSH key was Ed25519.

The harness's _run_ssh saw permission-denied exceptions, the wait loop caught them as transient failures, and kept retrying for the full 612-second window. The loop was working correctly. The diagnosis layer above the loop was wrong.

The actual diagnosis chain

The sequence after Run 12 reported the same failure:

1. Restored the baseline VM manually. SSH worked in 5 seconds. So the snapshot infrastructure was fine and the baseline state was clean.
2. Took a screenshot of the VM console. Showed "mission-app login:" prompt, fully booted, just no network — or so we thought.
3. Filed a hypothesis: "FIPS-mode breaks RHEL 9 NetworkManager-DHCP, which is a documented risk class per Red Hat." Started writing it up.
4. Searched the web for evidence. Found Red Hat's warning that post-install FIPS is unsupported, but no specific NetworkManager-DHCP-breakage CVE/bug. The search was building the case for the wrong root cause.
5. Probed the layers individually instead of trusting the SSH failure. Pinged the VM — succeeded. nc'd port 22 — succeeded. Ran ssh -vv — saw the actual permission denied.

Step 5 is where the diagnosis broke. The screenshot in step 2 told us "the VM is booted but unreachable." The ping in step 5 told us "the VM is reachable." The contradiction between the two should have happened in step 3 — but the harness's "no route to host" error from a previous ARP-pending state had primed us to believe the network was actually down. The first three steps were building a case from a stale ARP cache.

What the production fix was

Four commands. Fifteen minutes of actual work.

ssh-keygen -t rsa -b 4096 -N "" -f /data/vm/gemma-forge/keys/adm-forge-rsa
sudo virsh shutdown gemma-forge-mission-app  # ACPI shutdown of the running FIPS VM
sudo virt-customize -a /data/vm/gemma-forge/pool/mission-app.qcow2 \
    --ssh-inject adm-forge:file:/data/vm/gemma-forge/keys/adm-forge-rsa.pub
sudo virsh start gemma-forge-mission-app

Plus a symlink at the canonical key path:

cd /data/vm/gemma-forge/keys
mv adm-forge adm-forge.ed25519           # preserve the original
ln -s adm-forge-rsa adm-forge            # canonical path → FIPS-compatible key

The symlink approach means zero code or config edits in the repo. Every reference to /data/vm/gemma-forge/keys/adm-forge — and there are many (config/harness.yaml, bin/forge, gemma_forge/harness/loop.py, the CVE skill, the test suite) — automatically picks up the FIPS-compatible RSA-4096 key.

Then snapshot the working FIPS-enabled state as the new baseline, preserving baseline-prefips as the documented fallback for any operator who needs the non-FIPS state back.

What the FIPS-baseline does to the workload

A smoke STIG scan on the new baseline:

Result	Count	Δ vs pre-FIPS
pass	175	+7
fail	263	−7
notapplicable	46	(unchanged)
notselected	1031	(unchanged)

Six of the eleven FIPS-family rules now pass at scan time:

Pre-passes on FIPS-baseline	Still fails
fips_crypto_subpolicy	aide_use_fips_hashes (config-only edit)
sysctl_crypto_fips_enabled	enable_fips_mode (combined FIPS:STIG policy)
harden_sshd_ciphers_openssh_conf_crypto_policy	fips_custom_stig_sub_policy (combined FIPS:STIG policy)
harden_sshd_ciphers_opensshserver_conf_crypto_policy	configure_crypto_policy (combined FIPS:STIG policy)
harden_sshd_macs_openssh_conf_crypto_policy	grub2_audit_argument (kernel cmdline → DEF-29 territory)
harden_sshd_macs_opensshserver_conf_crypto_policy

The remaining 5 are clean-shaped: four are config-only edits the Worker can land without any destructive system mutation, and one is a clean single-item reboot family that exercises DEF-29 the way the architecture was designed to be exercised. No 7-item batch this time, no fips-mode-setup --enable script destroying the network. The reboot is the genuinely-architectural kind — adding a kernel command-line argument.

Predictions for Run 13

Same betting format as 38.10, slimmer because there's less to bet on.

1. Fix rate lands in [94%, 96%]. Confidence: high.

The 7-rule head start at scan time directly translates to 7 fewer rules the Worker has to land. R12 was 230/255 = 90.2%; R13 starts from 256 failing rules (263 minus 7 already-handled by DEF-29 rules that became scan-passes... wait, 263 is the scan count after FIPS, so the Worker starts with 263 to remediate). If R13's per-rule pass rate matches R12's (which was 230/255 = 90% ex-skip), R13 lands at roughly 237/256 = 92.6%. The bet predicts higher because: - DEF-28-deeper OVAL criteria continue to help (R12's first-attempt success was 95.2%). - The destructive fips-mode-setup --enable apply script never runs (the rule isn't failing anymore), removing the entire class of post-reboot breakage. - The single grub2_audit_argument reboot is genuinely DEF-29's happy path.

2. Escalations land in [8, 14]. Confidence: medium-high.

R12 was 24 (18 in-loop + 6 deferred-failed). R13's residual escalation set will be roughly: 4-5 environmental/policy rules that no architectural change touches (installed_OS_is_vendor_supported, fapolicy_default_deny), plus ~4 hard scanner-gap edges DEF-28-deeper doesn't fully close, plus 0-3 rules in the FIPS-tail (aide_use_fips_hashes, the FIPS:STIG combined policy rules) depending on how the Worker handles them.

3. DEF-29 fires cleanly on the kernel-cmdline family. Confidence: high.

A single-item family with a documented, well-understood reboot path. The reboot itself is a normal Rocky 9 reboot — the FIPS state is already in place, so this reboot is no different from a stock kernel-cmdline reboot, which CVE has been doing successfully for runs. Wait window of 180s is more than adequate (the FIPS-baseline came up in 40s from cold boot).

4. Wall clock lands in [4h, 5h]. Confidence: high.

R12 was 5.25h with a 612-second SSH wait that was wasted. R13 has 7 fewer rules to remediate (saved time), one clean ~60-second reboot (negligible cost), and the same DEF-28-deeper prompt overhead.

5. First-attempt success rate stays above 93%. Confidence: high.

R12 was 95.2%. The FIPS-baseline removes a few destructive-apply edge cases the Worker had to recover from. Should hold or improve.

What 38.13 should grade

Same rubric as 38.10 → 38.11 grading: 1. Did fix rate land in [94%, 96%]? 2. Did escalations land in [8, 14]? 3. Did DEF-29 verify the kernel-cmdline family cleanly? 4. Did wall clock land in [4h, 5h]? 5. Did anything in the FIPS-tail surprise us?

If 1, 2, 3 all yes, the four-run experimental arc (R8→R9→R10→R11→R12→R13) closes cleanly. STIG is at the architectural ceiling for the current paradigm — any further improvement requires DEF-30 (dynamic prompt enrichment), a richer Worker, or a different skill.

What this entry's lesson is

Two separate ones, ranked by how generally they apply.

1. When a tuning fix doesn't move the failure, the diagnosis was wrong. Not under-tuned. Not slightly miscalibrated. Wrong. The cheap heuristic: if you bumped a constant by 2.5x and the failure landed in the same place 2.5x later, the constant was not the cause — you're just walking through a wall slowly. Stop. Probe a layer below the symptom directly. Ninety seconds of ssh -vv would have caught this in Run 11, not Run 12.

2. Production-grade fixes are usually smaller than the production-grade story. This entry's title bills the work as a "production-grade pivot," and it earns the framing — FIPS-enabled gold image is the actual deployment workflow for STIG-compliant fleets. But the work itself was four commands and a symlink. The proportion of debugging-to-fixing was about 50:1. That ratio is the more honest reading of "production-grade": most of the work in production is figuring out which line to change. The change itself is rarely the hard part.

The four-run arc gets one more entry to close. Run 13 grades it.

---

Related

• journey/38.11 — the entry whose wrong-diagnosis this entry pivots away from.
• adr/0021 — DEF-29's engineering record. Gains a footnote about the SSH-key class of issue.
• config/harness.yaml — unchanged; the symlink at the canonical key path means the harness automatically uses the FIPS-compatible RSA key.
• Red Hat: Switching RHEL to FIPS mode — the doc whose post-install warning we accepted, then accidentally validated.
• Next: journey/38.13 after Run 13 lands. The arc closes there if the predictions land.