40.5. Phase 1 Lands Clean — The Intel Corpus Pipeline¶

Same-day continuation of entry 40. Captured here because the build went smoothly enough to be its own "yes, the proposal's architecture survives contact with code" data point. Worth recording the things that worked AND the small build-time decisions that don't rise to ADR-amendment-level but shape the next phase.

Why this is its own entry¶

Three things made Phase 1 worth recording separately rather than just collapsing into a future "the whole skill shipped" entry:

The dynamic-lookup architectural claim survived first contact with code. ADR-0023 §1 proposed shifting from a static curated playbook to a runtime CVE-DB query. Pre-flight measured pool size (75 entries at strict intersection). Phase 1 actually built and ran the query pipeline against 26,946 unified entries — and the ranking heuristic produced sensible results first try. The architectural pivot was load-bearing; it didn't fold under implementation pressure.
The Postgres-bootstrap → migrations → ingestion pipeline is now "well-paved." Third skill to use tools/bootstrap_skill.sh --skill pentest; third skill to template migrations from cve/; the per-skill schema isolation pattern (forge_pentest role + pentest schema) holds with no friction. This was a Phase D ADR-0016 bet that's paying off.
The refresh-cadence operational story is real. 17 seconds for the full multi-source refresh (CISA KEV + 10MB MSF metadata + 10MB ExploitDB + sparse Vulhub clone). That's well within the "run before every engagement" budget the architecture assumed. Operators won't push back on a 17-second startup step.

What Phase 1 actually delivered¶

26,946  unified CVEs in the corpus
33,243  exploit recipes (multiple per CVE — msf + ExploitDB fallback)
   244  Vulhub container scenarios indexed (CVE→path mapping)
 1,602  CISA KEV entries (federal "actively exploited" — 3 days fresh)
 2,497  msf-covered CVEs
25,003  ExploitDB-covered CVEs
    17  seconds for full refresh from scratch sources

The architectural claim from futures/pentest-skill.md:

"The agent isn't picking from a 75-entry menu. It's using nmap + a real-time CVE lookup + a real exploit tool + cross-run memory."

Phase 1 built the "real-time CVE lookup" half of that claim. The agent's Worker, when Phase 3 lands, will call lookup_candidates( service, version) and get back this:

lookup("apache", "2.4.49") →

  [KEV] CVE-2021-41773  rank=100.0  Apache/HTTP Server
        Apache HTTP Server Path Traversal Vulnerability
        → msf:exploit/multi/http/apache_normalize_path_rce  reliability=0.50
        → edb:exploits/multiple/webapps/50383.sh             reliability=0.30
        → edb:exploits/multiple/webapps/50512.py             reliability=0.30

That ranking is right: KEV-listed (federal credibility) + msf module available + Vulhub-reproducible + ExploitDB fallback PoCs → top of queue. The Worker iterates cands[0].recipes in reliability order and gets a working msf module first, fallback PoCs after.

Things worth recording from the build¶

1. The ranking heuristic ended up simpler than I expected¶

The Worker turn budget is small; we don't want a fancy multi-factor scoring model the agent has to interpret. A weighted sum capped at 100 turns out to be enough:

  KEV-listed                     +50  (federal credibility primary)
  Known ransomware campaign use  +15  (active threat)
  Has msf module                 +20  (working tool ready)
  Has Vulhub container           +15  (we can reproduce it in our lab)
  Has ExploitDB PoC              +10  (fallback tool)
  Published year ≥ 2020          +5
  Published year ≥ 2023          +5

A KEV-listed CVE with all four source signals lands at exactly 100. A non-KEV CVE with only an ExploitDB PoC lands at 10-20. The "try the high-confidence stuff first, fall back to long-tail" ranking emerges from the weights without needing complex logic.

This is a calibration choice that may want tuning after engagement data comes in (ADR-0023 §"Open questions" #5 names this). v1 weights are conservative-by-construction; the goal is "obviously right CVEs rank first," not "optimal triage."

2. The recipe-table split was the right call¶

Initially I considered storing one canonical recipe per CVE in the main cve_intel row. Walking through Worker behavior pushed against that:

Some CVEs have a Metasploit module AND multiple ExploitDB PoCs.
The Worker should be able to try the msf module first (high reliability, structured invocation), fall back to an EDB script (lower reliability but still works) if the msf attempt fails.
That "try in order, accept what works" pattern needs multiple recipes per CVE retrievable as a list.

cve_exploit_recipes as a separate table with reliability_score per row, ordered DESC when joined, gives the Worker a clean "iterate this list until one works" interface. Same shape as how a real pen tester would think: "try msf first, the script if that fails."

3. Fingerprints are explicitly NOT auto-generated¶

CISA KEV gives vendor + product, but not the regex/range patterns needed to match nmap banner output to a CVE. Auto-generating these from CISA data would be possible but inevitably noisy — wrong-version matches would have Ralph wasting attempts on non-vulnerable targets.

ADR-0023 §3 vetting policy: "We do NOT want Ralph wasting attempt budget on containers that are simply not in an exploitable state." The same logic applies to fingerprints. So Phase 1 leaves cve_fingerprints empty; Phase 2 populates it as part of target pool vetting, where every fingerprint is hand-verified against the actual Vulhub container's banner output.

This pushes labor to Phase 2 (the 2-day vetting estimate is mostly this), but it preserves the "Ralph never wastes attempts" property. Worth it.

4. UPSERT-based refresh is idempotent + incremental¶

Every CVE in the corpus has a last_refreshed_at and a last_source_set (which sources had this CVE in the most recent refresh). Re-running the refresh:

New CVEs added since last run → INSERTed
CVEs that left a source → still kept, but last_source_set drops that source (gives us visibility into "this CVE used to be in KEV but isn't anymore" — uncommon but happens)
CVEs in all sources → UPDATEd with current data + bumped last_refreshed_at

This means the refresh doesn't need a "drop and rebuild" step. It can run mid-engagement without disrupting in-flight Worker queries (though we don't currently do that — refresh is engagement-start only).

5. The `--skip-download` flag is a small but high-value developer-loop affordance¶

During Phase 1 development, I ran the refresh ~30 times. Re-pulling 20MB of source data each time would have been wasteful + would have made me reluctant to iterate. --skip-download reuses the existing scratch dir; iteration cycle becomes 2s instead of 17s, and download bandwidth is preserved for actual refreshes.

Small flag, real productivity multiplier. Worth noting for future similar tools.

What didn't surprise us, but is worth confirming¶

Multi-source de-duplication works cleanly via CVE-ID as primary key. Same CVE in 3 sources gets merged into one row with per-source fields populated separately.
Postgres ARRAY + JSONB types handle the recipe metadata cleanly. No need for separate join tables to N-many recipes; ARRAY columns on the parent + a JSONB option-blob on each recipe keep the schema flat enough.
The bootstrap script + apply_migrations.sh pattern Just Works for the third skill in a row. Phase D's per-skill schema isolation bet from ADR-0016 has paid back the original investment several times over by now.

What surprised me a little¶

The MSF metadata file's CVE references aren't always consistent. Some modules have references: [["CVE", "2021-41773"]] (list-of-pairs); some have references: ["CVE-2021-41773"] (list of strings). The parser handles both, but the schema inconsistency is genuine MSF quirk. Documented in corpus_refresh.py.
MSF's rank field is inverted from what feels intuitive. Lower number = higher reliability (100=Excellent, 600=Manual). The inversion is per MSF convention; the corpus normalizes to a [0,1] reliability_score where higher = better. One-line lookup table; trivial to handle but easy to get backwards.

The Phase 2 path is now well-defined¶

Phase 2 is the hand-vetting phase, and Phase 1's design pushed all the vetting-dependent state (fingerprints, accurate recipe metadata) into tables Phase 2 will populate. So Phase 2's deliverables are unambiguous:

For each target in the curated pool: an entry in target_pool (vetting_status = plug_and_play or needs_setup with script).
For each CVE represented by a vetted target: matching entries in cve_fingerprints (banner patterns that nmap actually returns for that container) and refined cve_exploit_recipes rows (correct msf option names, verification config tuned to what the exploit actually does, prerequisite_setup_script where needed).

Two days of focused vetting work. ~60 containers, ~10-15 minutes each (per ADR-0023's vetting cost estimate), aligned with the build-sequence preflight doc.

Recap and what's next¶

Phase 1 took ~3 hours real time end-to-end:

30 min: ADR-0023 lock-in updates (the four pre-build decisions)
30 min: schema bootstrap + migrations applied
1.5 hours: corpus.py + corpus_refresh.py + shell wrapper
30 min: end-to-end smoke + commit

Half a day. Phase 2 (target pool vetting) is multiple days because the labor is per-container not per-script. Phase 3 (skill code) is ~3 days of code with clear deliverables. Phase 4 (cross-run measurement) is 1 day of run + analysis.

Total against the ~2-week estimate: still on track. The pre-flight scope estimate is holding so far.

Commit landed at 2719a36 on branch feat/pentest-skill-proposal. Phase 2 starts immediately after this entry lands.