Gotcha: five things smokes caught in Phase 3 of the pen-test skill

Phase 3 of the network-pentest skill was built across six sub-phases. Every sub-phase ended with a smoke script that exercised the new code against a live containerized target. Every single smoke found a real bug — and four of the five forced architectural changes rather than local fixes. Worth recording because each one is the kind of thing that's invisible until you actually run the code against a real adversarial environment.

1. Fingerprint-only corpus matching doesn't scale to 27k CVEs

Symptom

Phase 2.6 smoke pipeline ran 5/5 targets end-to-end (spin up → nmap → corpus query → tear down). All 5/5 returned zero CVE candidates.

Root cause

CorpusClient.lookup_candidates() matched only against cve_fingerprints (13 hand-vetted entries). The intel corpus had 26,946 CVEs and 33,243 recipes, but the lookup ignored all of them unless a hand-vetted fingerprint pre-existed. Workable for a 50-CVE curated playbook; impossible for a corpus this size.

Fix

Two-tier lookup with a 0.5× rank multiplier:

1. Precise fingerprint match (rank ×1.0)
2. Vendor+product fuzzy match against cve_intel directly (rank ×0.5)

Vendor-fallback hits get demoted but still surface. Now 4/5 targets return candidates including exact-CVE recovery on WebLogic (CVE-2017-10271 from blind nmap banner).

Generalization

A corpus is structurally different from a playbook. A playbook can expect every entry to be reachable by a hand-built key. A corpus needs fuzzy fallback layers because no one will hand-vet 27k entries. Plan the lookup layers when you plan the corpus shape; don't bolt them on later.

2. Vendor-alias fallback over-matches when the service token is generic

Symptom

Phase 3.1 smoke. nginx 1.14.0 target returned Apache HTTP Server CVEs. The corpus has zero nginx CVEs with recipes; the fallback should have returned empty, but it returned Apache via the generic "http" service-name alias.

Root cause

lookup_by_banner tries multiple aliases in priority order (product_token → vendor_token → raw service). When nginx returned empty (correct), it fell through to "http" (incorrect), which matched every Apache product via wildcard.

Fix

Known-only restriction: if ANY alias resolves to a known service in _KNOWN_SERVICE_ALIASES, restrict the search to those known aliases. Don't fall through to generic terms like "http" or "ajp13".

Generalization

Fallback layers must terminate at vendor-level specificity, not at "any string that matches anything." The most-specific alias that returns nothing should fail with empty — not silently search broader. That's how false positives bloom in corpus-driven systems.

3. msfconsole isn't on $PATH in non-interactive docker exec

Symptom

Phase 3.3 MsfEngine smoke. Subprocess exit code 127, error:

OCI runtime exec failed: exec: "msfconsole": executable file not
found in $PATH

Root cause

The metasploitframework/metasploit-framework image sets PATH in its entrypoint, not its base image config. When you docker exec the container with a fresh process, the entrypoint hasn't run for THIS process — you get the base image's PATH, not the entrypoint's augmented one. Tools the image documents as "available" are unreachable by name.

Fix

Use the absolute path: /usr/src/metasploit-framework/msfconsole.

Generalization

Tools installed by a container image are not necessarily on $PATH for docker exec. If you're scripting against a container that documents tool X, find the absolute path before assuming exec finds it. The interactive shell where you tested manually is a different process than docker exec will spawn.

4. Banner-based recon can hide the actual vulnerable component

Symptom

Phase 3.4 runtime smoke. Target target-php-cve-2012-1823 (vulhub PHP-CGI). nmap -sV returns:

80/tcp open  http    Apache httpd 2.4.10 ((Debian))

Corpus correctly returned low-confidence Apache HTTP Server CVEs (rank=50 via vendor wildcard). None applicable. The actual vulnerable component — php:5.4.1-cgi running behind Apache — wasn't in the banner at all.

Root cause

nmap fingerprints the OUTERMOST server. PHP-CGI runs behind Apache; the Apache wrapper answers the TCP handshake. Banner ≠ vulnerable component. The corpus + lookup were correct; the assumption that banner identifies the vulnerable component was wrong.

Fix

Surface target_pool.intended_cve_ids recipes as a separate pool_intel block in worker_context() alongside the banner-derived candidates block. Worker prompt: "try banner candidates first; fall back to pool_intel when those fail." Realistic — real pentest engagements get threat-intel briefings about what's known to be running BEFORE recon.

Generalization

Discovery and identification are different concerns. Recon tells you what's listening on the wire. What's actually vulnerable may be further down the stack and invisible to the banner layer. Build the architecture to accept multiple identification sources — banner + prior intel + (eventually) Reflector tips that name misleading banners — and let the agent pick.

5. Module-level state in importlib.module_from_spec modules is orphaned from sys.modules

Symptom

Phase 3.6 skill smoke. The Worker tool propose_exploit returned:

ERROR: no current target — harness state corrupt

Even though we'd just pushed a current item.

Root cause

The harness's _build_skill_runtime uses importlib.util.module_from_spec(...) + spec.loader.exec_module(...) to load runtime.py dynamically. Neither of those calls inserts the module into sys.modules. The class lives in an orphan module; sys.modules[type(runtime).__module__] raises KeyError. Any external caller wanting to access the runtime's module-level state (_current_item, _db_conninfo, etc.) can't find the module.

Fix

Don't push state via sys.modules lookup. Add an instance method on the runtime class — set_current_item(item) — that knows it's in the same module as the state and mutates it directly. Callers use the setter; module discovery is no longer needed.

Generalization

Module-level state in dynamically-loaded modules needs an instance-method entry point. If your protocol allows the loader to skip sys.modules registration (Python's importlib does), external callers can't find the module by name. The class can always find its own globals; expose a method.

Process meta-observation

None of these five would have surfaced from unit tests. They required:

• A real Docker container running on a real network
• A real corpus with real false-positive patterns
• A real msf binary in a real image with real $PATH quirks
• A real vulhub stack with the actual layered-services anatomy
• A real harness loader that exercises the actual code path

The general lesson: for systems that interact with real adversarial environments, smoke-driven implementation beats test-first. See architecture/03-smoke-driven-implementation for the pattern that surfaced these.