Skip to content

40.7. Phase 3 — The Skill Stands Up Without an Agent in the Loop

Continuation of 40.6. Phase 2 closed with a working pool + corpus and a real architectural finding (vendor-fallback over a 27k-CVE corpus is structurally different from a curated playbook). Phase 3's job was to build the skill itself: the runtime that sits between the agent's LLM calls and the actual containers being exploited.

The estimate was 3 days. It took one. The architecture survived first contact with code so cleanly that "validation smokes" kept passing on the first or second try.

What was built, in six commits

Sub-phase What landed Commit
3.1 recon.py (reusable HTTP-probe fallback) + 11b3f8a
lookup_by_banner (banner-aware corpus lookup)
3.2 6 verification primitives + 25/25 unit tests + c9cdd0a
DB CHECK constraint on verification_class
3.3 3 exploit engines (msf / exploitdb / curl) + 3be2729
driver container + lazy-fetch from EDB GitLab
3.4 4 sub-runtimes + worker_context() + pool_intel 258416d
3.5 5 prompts (architect/worker/auditor/reflector/judge) 006c931
3.6 skill.yaml manifest + full harness wiring smoke 9a0798e

The thing that worked first try

The Phase 3.3 engine smoke achieved real RCE on a real target on first execution of the MsfEngine path:

[3/3] MsfEngine (php_cgi_arg_injection)
[PASS] MsfEngine + session_opened (engine_ok=True, verified=True, elapsed=22.7s)
       verify: new meterpreter opened against 172.27.0.2:4444

The agent didn't have to retry, didn't have to consult tips, didn't have to be told what payload to pick — the engine took a corpus recipe row, built a resource script, ran msfconsole -q -r, parsed the session-open line out of the output, fed the observation into verify.verify('session_opened', ...), and returned a clean PASS.

That's the exploit loop the whole skill is built for. End-to-end. No LLM in the loop. Twenty-two seconds from "spin up vulhub container" to "meterpreter session opened against this host."

This isn't a demonstration. It's the actual mechanism Ralph will use to learn. Phase 4 is going to add the LLM that picks WHICH recipe to try and WHEN — but the verification end of the loop is already binary and trustworthy.

The architectural finding worth recording

Phase 3.4's runtime smoke surfaced a real-world wrinkle the corpus design didn't fully anticipate: banner-based recon can under-resolve the actual vulnerable component.

The php/CVE-2012-1823 target's nmap -sV returns:

80/tcp open  http    Apache httpd 2.4.10 ((Debian))

Which is correct — there is an Apache 2.4.10 wrapper. But the vulnerable component is php:5.4.1-cgi running behind it, and the corpus's banner-driven lookup_by_banner("http", "Apache httpd 2.4.10") correctly returns Apache HTTP Server CVEs (with the proper 0.5× vendor-fallback multiplier showing low confidence) — none of which are the actual exploitable one.

Fix landed in 3.4: worker_context() now returns TWO blocks:

  • candidates — banner-derived CVE candidates (what a fresh recon pass would tell you)
  • pool_intel — recipes for the target's intended_cve_ids from target_pool metadata (what threat intel would tell you BEFORE the engagement)

The Worker prompt explicitly tells the agent: "try banner candidates first; fall back to pool_intel when those fail." This is realistic — real pentest engagements get briefed on what's known to be running before the consultant touches a keyboard. The agent's job isn't to re-discover the CVE; it's to pick the right recipe and option set, and that's still hard enough to be interesting.

ADR-0023 §3's "iron-clad binary ground truth" claim was about verification, not target identification. The verification end stayed binary throughout (six primitives, 25/25 tests). Target identification is multi-source: banner + pool intel + (future) Reflector tips that say "for this CVE, expect this misleading banner."

This is the kind of finding that's worth recording because it would have surprised the proposal author. It didn't break the architecture — the corpus + verify infrastructure still works exactly as designed — but it changed what worker_context carries.

Build-time numbers

What we now have (pentest schema):

  • 26,946 unified CVEs in cve_intel
  • 33,243 exploit recipes in cve_exploit_recipes (CHECK-constrained to 6 verification classes)
  • 13 hand-vetted entries in cve_fingerprints (the precise-match path)
  • 20 plug_and_play targets in target_pool
  • 17 seconds for a full corpus refresh
  • 7 seconds for cold target spin-up (vulhub PHP-CGI)
  • 22 seconds for an msf RCE round-trip
  • 0.5 seconds for a CurlEngine one-shot inline RCE
  • 25/25 verification-primitive unit tests pass

The first measurable Worker turn — the one that matters — will take the form: agent picks (cve_id, recipe_id, target_port, options) → propose_exploit fires → engine runs → verify returns binary → Evaluator captures → Reflector reads on fail. The whole chain is green-light on the runtime smoke.

What's NOT done yet

Phase 3 is the structural skill. Three things wait for Phase 4:

  1. Run 1 cold — actual LLM Worker on the live skill, no tips in memory yet. We'll see which targets the agent can exploit blind, which take multiple attempts, and where the Reflector earns its keep.

  2. Dream pass — read Run 1's attempts, judge which retrieved tips were actually followed, score each by outcome, evict low-utility tips. The graphiti-backed memory plumbing already exists in the harness from prior skills.

  3. Run 2 warm + cross-run measurement — different random target shuffle (the shuffle_seed pattern makes this reproducible) + memory-loaded. Headline metric: PASS rate Run 1 → PASS rate Run 2. ADR-0023 §4 locked +20pp as the success threshold. If we hit it blind without target memorization (different random sample each run), cross-run learning is empirically real on this skill.

How this changes the project narrative

Three skills shipped before this one. STIG (Run 14: 90.3%) and CVE remediation both demonstrate the architecture but speak to specific verticals. Detection-tuning shipped functionally but didn't produce a clean result because the corpus labels were noisy and the PASS criterion was misaligned with reality. We pivoted away from it.

Pen-test is the first skill that's both:

  • Architecturally distinct — different corpus shape (multi-source intel union), different target shape (container pool with random rotation), different verification shape (six binary classes), and the FIRST skill where the agent reads from a corpus that REFRESHES PER ENGAGEMENT instead of being curated upfront.
  • Defensible — federal-grade sources (CISA KEV, NIST NVD, MITRE ATT&CK, Metasploit, ExploitDB, OSV, GHSA), iron-clad binary ground truth (no F1 fuzziness), and the cross-run measurement built into the run shape itself (random target rotation makes "memorization" structurally impossible).

If Phase 4 hits its measurement target, the project has shipped its strongest evidence yet for cross-run learning as a real phenomenon — not just an artifact of repeated exposure to the same items.

That's what Phase 4 is for.