Skip to content

40. Back to the Drawing Board — the Fourth Skill Becomes Autonomous Pen-Testing

This entry covers a 4-hour window: detection-tuning's Run 2 finishing its last escalation (0/5 PASS, same as Run 1), the "this result doesn't stand on its own" call, the wider-aperture brainstorm, the pivot to pen- testing, and the architectural sharpening that came from external pushback. By the end of the window we had three docs landed (pentest-skill.md, preflight, ADR-0023) and a clean decision gate.

Why this is its own entry

Three things converge here that haven't converged before in the project:

  1. A skill ships functionally but doesn't produce a standalone result. That's a new outcome category. STIG and CVE both shipped AND landed as clean results. Detection-tuning shipped but the result was 0/5 PASS twice. Captured honestly: working architecture + non-compelling headline = "set aside, not torn down." Same lesson the v1→v2 memory rewrite captured at the architecture level, now at the skill level.

  2. The construct's distinctive properties get named explicitly and used as the selector. The prior skills exploited 2-3 of {patient grind, cross-run memory, sovereignty, token-as-input}. The pen-testing pivot is the first time we ASK "where do all four matter simultaneously?" as the gating question for the next skill. That question changes the brainstorm shape — and the answer.

  3. External pushback shaped the architecture, not just the scope. Three pushbacks ("if we plant the vulnerability, it's theater" / "we'd want the latest intel, not a one-time download" / "is the corpus wide enough? AI can pull from more sources than an analyst") each changed a load-bearing architectural decision. The skill ended up structurally different from the first sketch in a way that's directly traceable to those three exchanges.

What Run 2 made unambiguous

Detection-tuning's Run 2 against the SDS corpus (warm with Run 1's 90 tips loaded) closed at 20:33 with 5/5 escalated, same as Run 1. Wall clock was 1.8× faster — the Architect used prior-run history to escalate dead-end rules sooner — and the first rule the agent picked hit F1=0.222 on attempt 1 against a cold SDS baseline of 0.12, a 1.85× improvement on the first measurable opportunity for cross-corpus transfer.

So the architecture worked. Tips persisted, dream pass ran, cross-corpus transfer happened, the Architect's escalation logic sharpened with prior data. Every ADR-0022 load-bearing claim held.

But "0/5 PASS in both runs, the second run was faster" isn't a result that stands on its own.

Two compounding root causes, both downstream of the original skill framing:

  • The PASS bar. P >= 0.95 AND R >= 0.80 was lifted from generic- ML evaluation. Real detection engineers don't tune to it; they tune to "false positives per N hosts per day" against production telemetry. With the corpora we used, no rule could reach that bar because the achievable F1 ceiling sat around 0.7-0.75 — a labeling problem, not a tuning problem.
  • The corpus labels. EVTX-ATTACK-SAMPLES and SDS both label at file level ("this capture demonstrates mimikatz"), but each capture contains hundreds of routine system events alongside the attack-specific ones. My loader marked them all positive. That labeling fuzz capped recall regardless of how good the Worker was.

The honest framing: detection-tuning proved the harness can run a graded-outcome skill end-to-end, but the tractable version of that problem isn't autonomously solvable against publicly-available academic corpora with a strict generic-ML PASS bar. Captured the full disposition in entry 39.

"Back to the drawing board"

The user's words. Within an hour of the Run 2 final escalation, the question was no longer about detection-tuning's tuning — it was about what the fourth skill should be such that it actually lands.

The first brainstorm pass listed STIG-shaped candidates:

  • Kubernetes pod failure remediation (CrashLoopBackOff → diagnose → fix Deployment YAML)
  • CI / build failure auto-fix (CI red → diagnose → push fix → CI green)
  • Cloud Security Posture Management (CSPM scanner findings → remediate → re-scan)
  • Database performance tuning (slow queries → propose indexes → measure)

All reasonable. All structurally STIG-shaped (binary outcomes, scanner or test result is the ground truth, fix is a config or code edit).

The user pushed wider. Their words, roughly: "let's go wider. Think about the construct — long-running agent at the edge (or core), the ability to learn from its mistakes, gets better over time, the ability to throw raw tokens at a problem until we find the solution. Let's come up with something compelling."

That reframe changed the selector. The four construct properties named together:

  • Patient grind: hours/days on one problem, instead of one-shot
  • Cross-run memory: tip pool that compounds across engagements
  • Sovereignty: runs on-prem, on the operator's own hardware, data never leaves
  • Token-as-input: willing to burn millions of tokens to find an answer

The four-property test made the list short. Where do ALL FOUR matter simultaneously, not just two or three? K8s pod failure: 2-3 (no sovereignty necessity, no real patience requirement). CSPM: 2-3 (STIG-shaped, no sovereignty hardness). Most candidates: 2-3.

Pen-testing: all four. A long engagement against an unknown network needs patience (technique trees go deep), memory (the target environment teaches the agent about itself), sovereignty (no operator runs autonomous exploit tooling via SaaS LLM into a classified network — full stop), token-as-input (a human red teamer at $200/hr is $2400 over a 12-hour engagement; Ralph at the same wall-clock is $20 of inference).

That was the gate. Pen-testing was the answer.

Pushback 1 — "if you plant it, it's theater, not engineering"

My first sketch: "we deploy intentionally-vulnerable VMs and have Ralph find what we planted." The user nailed the credibility problem in one sentence: "you knew what kind of hole you created. This is only really interesting if it can pull from somewhere a list of vulnerabilities to try... it has to truly work, not work as a setup."

Correct. "We made a hole, agent found it" isn't engineering.

The legitimate version is: agent gets the same intel a real pen tester has (public CVE database, published exploits, Metasploit modules, ATT&CK techniques) and works against a target it has no privileged knowledge about. If it finds something, the finding is real.

The pivot from "planted vulnerability" to "use third-party-built vulnerable lab targets + public intel" came directly from this pushback. Vulhub became the target source (244 community-built CVE scenarios; MIT licensed; used in federal cyber training curricula). Public intel sources (CISA KEV, NIST NVD, Metasploit, ExploitDB) became the agent's knowledge base.

Pushback 2 — "we'd want the latest, not a one-time download"

My second sketch: "hand-curate a static playbook of 75 high-confidence CVEs with exact Metasploit module + payload + verification per entry." The user nailed the second problem: "a true penetration testing Ralph loop wouldn't download the vulnerabilities once and use that forever. It would want the latest."

Also correct. A real 2026 pen-test tool deployed into an operational environment must have current intelligence. CVEs published this week must be available; CISA KEV adds entries continuously.

The pivot from "static curated playbook" to "dynamic per-engagement intel refresh" came from this pushback. The architecture became:

  • Per-engagement boot (~5 min): refresh CISA KEV JSON, msf metadata, ExploitDB CSV, Vulhub git HEAD; rebuild queryable index in Postgres pentest.intel_corpus.
  • Per-run (Worker loop): nmap target → query corpus lookup_cves(service, version) → ranked candidate list → try in order → record outcome.

The agent's intelligence responsibility narrows to pattern-matching + parameter-filling + tool orchestration — exactly within Gemma 4 31B's reliable competence window. The "novel exploit synthesis" capability the model doesn't have is never required.

Pushback 3 — "is our corpus wide enough?"

After I named four sources (CISA KEV, Vulhub, Metasploit, ExploitDB), the user pushed: "is our corpus wide enough? it's one of the benefits of using AI, we can pull from more sources than a normal analyst could."

Right again. I'd left out MITRE ATT&CK (technique taxonomy), OSV.dev (open-source ecosystem advisories — npm/PyPI/Maven/Go), GitHub Security Advisories (CVEs in OSS with linked patch commits), Vulners (commercial aggregator). The pivot from "four-source corpus" to "multi-source union" came from this pushback.

The credibility chain landed iron-clad: every source is federally-credible. CISA's own catalog. NIST's CVE structured data. MITRE's ATT&CK taxonomy (MITRE is an FFRDC). Metasploit (federal red team training standard). ExploitDB (the OSCP cert publishers). Vulhub (used in CISA's own training curricula).

The narrative writes itself: "We're testing the exact CVEs CISA tracks as actively exploited, using the exact tools every OSCP-certified red teamer uses, in a lab environment of the exact container scenarios used in federal cyber training programs."

Pre-flight, in 4 hours

Three checks against the new architecture:

Pre-flight 1 — corpus size & freshness. Pulled CISA KEV (1,602 entries, 3 days fresh), Metasploit metadata (2,497 exploit-module CVEs), Vulhub (244 docker scenarios), ExploitDB (25,003 CVEs with PoCs). Cross-referenced: 75 CVEs at the strict intersection of KEV ∩ Vulhub ∩ (Metasploit ∪ ExploitDB). Famous-name CVEs throughout — Shellshock, Heartbleed, SambaCry, Log4Shell, Spring4Shell, Drupalgeddon2, Drupalgeddon3, Ghostcat, ImageTragick. 32 unique vendors. 11 vendor groups with ≥2 CVEs — built-in same-vendor pairs for measuring cross-run generalization within a vendor family. Green.

Pre-flight 2 — tool-chain tractability. Walked the full chain manually against a real Vulhub container. Lab Docker network up, target up on the network, msf container on the network, nmap from msf to target → version banner → msfconsole use exploit/... ; exploit. Encoded PHP webshell payload was visibly sent to the Drupalgeddon2 endpoint in the URL parameters. The chain works.

Caveats surfaced and captured:

  1. Some Vulhub containers need setup before exploits work (Drupalgeddon2 needs install wizard completion before its exploit endpoint responds).
  2. Metasploit module option names vary per module (ActiveMQ uses HttpUsername, not USERNAME). Per-CVE exact options must be in the corpus.
  3. Payload compatibility per target matters (ActiveMQ rejects cmd/unix/generic).
  4. Session-based exploits require msf to stay alive across exploit- fire and verification steps — implies msfrpc daemon architecture, not one-shot msfconsole -x.

All four are engineering signal, not blockers. ADR-0023 §1 and §4 integrate them into the architecture.

Pre-flight 3 — federal credibility chain. Every public source is federally-credible (catalog above). The narrative is bullet-proof. Green.

ADR-0023 captures seven decisions

The architectural decisions landed at proposal-time — same upfront discipline that ADR-0020 and ADR-0022 imposed for prior skills:

  1. Dynamic runtime CVE lookup, not static curated playbook — pivot from Pushback 2.
  2. Multi-source corpus union, not single source — pivot from Pushback 3.
  3. Containerized lab network with per-run random target rotation — forces honest cross-run generalization measurement (no caching possible, since Run 2's targets are disjoint from Run 1's).
  4. Metasploit RPC daemon (msfrpcd), not one-shot msfconsole — captured caveat from pre-flight 2.
  5. Iron-clad binary ground truth per work item — got artifact (session, command output, file read) or didn't. The fuzzy-labels trap that defeated detection-tuning is structurally absent here.
  6. Per-engagement intel corpus refresh — operational architecture from pushback 2.
  7. Cross-run learning measurement built into the experiment design — random target rotation between runs is the falsifiability mechanism.

What we measured that detection-tuning couldn't

The single biggest architectural improvement vs the prior skill is ground-truth quality.

Detection-tuning used labeled corpora where labels were applied at the file level ("this capture demonstrates mimikatz") and our loader treated every event in the file as positive. That labeling fuzz imposed a hard recall ceiling that no Worker prompt could break. The cross-run improvement signal existed (Run 2 attempt 1 hit 1.85× cold baseline F1) but was swamped by the noise floor.

Pen-testing has iron-clad binary ground truth: got a Metasploit session (verifiable via msfrpc), got command output matching expected pattern, or got the contents of a known file. No interpretation. No fuzz. The cross-run measurement (Run 2 first-attempt success on disjoint targets vs Run 1 cold first-attempt baseline) becomes a falsifiable claim with no labeling-noise floor.

The honest cost

~2 weeks of focused work, broken down:

  • 3 days: intel corpus pipeline (refresh scripts + queryable index)
  • 2 days: target pool curation (30-50 vetted Vulhub containers, per-container quirk encoding)
  • 3 days: skill code (5 sub-runtimes + msfrpcd wrapper + lab network management)
  • 1 day: first Run 1 + dream pass + Run 2 + cross-run measurement

Not weekend-trivial. STIG was 1.5 days end-to-end; CVE was 1 day; detection-tuning was 3 days of building + days of running. This skill is genuinely bigger because the target lab is non-trivial and the intel corpus is multi-source.

Captured up front so the commitment is honest, not creeping.

Where we are at the close of this window

Three docs landed on feat/pentest-skill-proposal:

Pre-flight is GO. Architecture is documented. Cost is honestly priced. Decision gate is commit ~2 weeks vs hold — a scoping decision, not a technical one.

The two distinct lessons from this window

Architecture-side: the construct's distinctive properties (patient grind + cross-run memory + sovereignty + token-as-input) work as a selector for "what skill should we build next." Using that test explicitly turned a wide brainstorm into a short list of one.

Process-side: external pushback shaped THREE load-bearing architectural decisions (planted-vs-real, static-vs-dynamic, single-vs-multi-source). Each was correctable in real-time because the architecture wasn't already built. Detection-tuning landed without that pushback cycle on its load-bearing decisions and shipped with a corpus + PASS bar that no amount of skill-side engineering could overcome. The lesson: if the first architectural sketch isn't pushed-back-on by someone outside the build loop, the build is strictly worse than it could have been.

What's next is straightforward. If the build commitment lands, the next entry is the build-narrative arc (Phase 1 → Phase 4) and the cross-run measurement headline. If the commitment holds, this entry stands as the captured-pivot decision point — same shape as the prior "pivot moments" the project has documented (CVE pivot was entry 33; this is the same structure for the pen-test pivot).