Vulnerability assessment vs penetration testing: what two scanners found in 27 known flaws

We use "vulnerability scan" for the automated tool run and "vulnerability assessment" for the wider job of listing and rating weaknesses, which usually starts with a scan. We tested the scan part: two scanners, no person steering them. The figures below describe scanners, and say nothing about a full assessment that adds a human review of the results.

The test apps are the 27 fixtures in our web-security-fixtures repository (commit 2c4a833). Each one is a small local web app with one known flaw and a fixed twin. We ran OWASP ZAP 2.17.0 and Nuclei v3.11.1 against them on 11 November 2026, inside a sandbox that allows loopback traffic only.

Results by setup: 6 setups, 27 flaws

We ran six setups. "Seeded" means the scanner got every route of the app and a low-privilege token. "Cold" means it got only the root URL. "Found" means the tool named the flaw class the fixture was built to show. "Partial" means it reported a related symptom, such as a missing header, but not the flaw itself.

The best single setup was the ZAP active scan on the seeded app: 3 flaws found, 1 partial, 23 missed. Across all six setups, 4 flaws were found at least once: HSTS, the HttpOnly cookie flags, the open redirect and clickjacking. Two more were found in part: CSRF and GraphQL. The other 21 were missed by every setup.

The row-level security fixture is a PostgreSQL setup with no HTTP surface. A web scanner has nothing to aim at, so we count it as a miss in every row.

Results by fixture

Each fixture name links to the post that reproduces that flaw by hand. The columns show the seeded runs, the best case for each tool.

Two cells need a note. The Nuclei security-txt template reports that a security.txt file exists. It fired on the broken server and on the fixed one alike, so we graded it as missed. The Nuclei open redirect hit in the cold run came from a template for CVE-2018-14474 (an old forum product) that happens to match /login?next=. The default seeded Nuclei run did not find the open redirect, and the -dast run did.

What the scanners did well

Both tools found the flaws they exist to find. A missing Strict-Transport-Security header, a cookie without the HttpOnly flag, a page that can be framed and an unvalidated redirect target are all visible in a single response or a short probe, and each tool reported them.

Seeding mattered. ZAP found the cookie flags only after we sent the login request, because its cold run saw a 404 at / and nothing else. ZAP found the open redirect only in the active scan, not in the baseline report. In the cold runs Nuclei found 3 flaws (HSTS, clickjacking and the open redirect) and ZAP found 1 (HSTS).

A scanner also checks things nobody repeats by hand on every release: server banners, missing headers on every page and known CVEs in off-the-shelf software. We built the fixtures ourselves, so they hold no known CVEs, and this test says nothing about how well the tools match them. That matching is what the Nuclei template library is built for.

What scanners miss and why

A scanner sends requests and judges the responses. It can tell that two responses differ. It cannot tell which response the app should have sent, because it has no copy of the rules. Every miss below comes from that gap.

Authorization. Six fixtures break an access rule: api-authorization-testing, idor-vulnerability, broken-access-control, tenant-isolation, owasp-api-top-10 and mass-assignment. No setup found any of them. In api-authorization-testing we gave both tools Bob's token and the URL of Alice's note. The server returns Alice's note to Bob, and nothing in that response marks the note as someone else's. Finding that needs two users and a rule about who owns what.

Login and session logic. Five fixtures break a step in a flow: 2fa-bypass, credential-stuffing, pkce, session-fixation and api-rate-limiting. These flaws sit in the order of steps and the limits between them, such as a password-only session that can already read the account, a rate limit keyed on X-Forwarded-For, or a session id that survives login. No setup found any of them.

Race conditions and business logic. The race-condition fixture has a single-use coupon that two simultaneous requests can both redeem. One request at a time looks correct, and neither tool reported anything. mass-assignment, tenant-isolation and credential-stuffing share the trait: the request is valid, and the flaw is that the app allows it.

Injection and file handling. sql-injection is a class scanners are built for, so we checked the miss by hand. ZAP sent 1,290 requests and reported nothing. A manual request with q=zzz' OR 1=1 -- dumped all rows. We did not run a dedicated injection tool such as sqlmap, so this result applies to ZAP and Nuclei only. path-traversal, zip-slip, host-header-injection and redos were also missed by every setup. timing-attack needs many timed requests compared statistically, and for security-txt the Nuclei template only reports that the file exists.

Blind SSRF. ssrf needs a callback server that the target can reach, so the tool can see the request arrive. Our sandbox blocks all non-loopback traffic and we started Nuclei with -ni, which turns off its callback service. Read the SSRF row as "could not confirm here", and not as proof that the scanners cannot find SSRF.

These are the flaws that a penetration test is built around. A tester signs in as two users, tries one user's token on the other's objects, repeats a request in parallel, reads the business rules and builds a request that the app should refuse.

Noise and false criticals

On the seeded runs of the 26 fixtures with a web surface, ZAP's active scan raised 84 distinct alerts (counted per fixture), and 79 of them were not about the flaw the fixture was built to show. Nuclei printed 691 result lines. They cover 288 distinct template and matcher pairs (counted per fixture), and 281 of those were off-class. Much of it is low or informational: version banners, missing optional headers, technology detection. On the hsts fixture, Nuclei also matched five JARM fingerprint templates named after command-and-control frameworks, for example mythic-c2-jarm, at info severity.

Two results were rated as serious and were wrong. On 2fa-bypass, Nuclei reported chamilo-lms-sqli at critical severity and ZAP reported Spring4Shell at high severity. The fixture has no SQL and no Spring. Both appeared on the vulnerable server and on the fixed one. Someone has to catch results like these before anyone files a ticket.

Scan or penetration test

Run a scanner often. It is cheap to repeat, it catches headers, cookie flags, exposed versions and known CVEs, and it fits in a release pipeline. Treat its output as a list to triage.

Commission a penetration test when the app has logins, roles, money or data that belongs to more than one customer. Those are the places where the scanners missed the most. A tester also reads the scanner output and drops the false criticals, so you get findings you can act on. Our scope pages describe the work: web application penetration testing and penetration testing.

How we ran it

Tools: OWASP ZAP 2.17.0 (the aarch64 macOS release) and Nuclei v3.11.1 with templates v10.5.1, on a Mac, with fixtures from commit 2c4a833 of the fixtures repository and Python 3.13. We updated the Nuclei templates once with network access. The scans ran with none.

Every job ran under sandbox-exec with this profile, so no scanner packet could leave the machine:

(version 1)
(allow default)
(deny network-outbound)
(allow network-outbound (remote ip "localhost:*"))
(allow network-outbound (remote unix-socket))

For each fixture we ran three configurations: a cold run on the vulnerable server (root URL only), a seeded run on the vulnerable server, and a seeded run on the fixed server. Each ran ZAP and Nuclei, and the seeded runs added Nuclei with -dast. For the 26 fixtures with a web surface that makes 208 jobs, and all 208 finished.

ZAP ran from an Automation Framework plan. The plan sent the seed requests, spidered for 1 minute to depth 5, waited for the passive scan, wrote the baseline report, ran the active scan with the default policy, and wrote the active report. The "baseline" row is that first report, not the zap-baseline.py script. The launch command was:

zap.sh -cmd -silent -port PORT -dir TMP/zaphome -config database.recoverylog=false -autorun plan.yaml

The active scan job used these limits:

activeScan: maxScanDurationInMins=6, maxRuleDurationInMins=1, threadPerHost=3

Nuclei ran with the default templates:

nuclei -l targets.txt -duc -ni -jsonl -o nuclei.jsonl -timeout 5 -rl 200 -c 10 -no-mhe
# seeded runs add:  -H "Authorization: Bearer TOKEN"
# the dast setup adds:  -dast

The seed for a fixture is the list of its routes with sample requests and one low-privilege token. We graded each tool's output with regular expressions on the ZAP alert name and the Nuclei template id, name and matcher. The CSV lists every alert and template that fired, so you can regrade with your own rules.

Caveats

• We adjusted the grading rubric after seeing the output. We wrote a first version before the scans and changed some patterns once we had read what the tools printed. The CSV keeps the raw alert and template names so you can check each grade.
• The active scan had a 6-minute cap and a 1-minute cap per rule. A longer run could find more.
• The fixtures serve one connection at a time, so one idle scanner connection stalled the whole app. We ran them with a threaded launcher that swaps in a threaded HTTP server and counts requests. The flaw logic is unchanged. For the HSTS fixture, the launcher also moves the TLS handshake into the request thread.
• The row-level security fixture has no HTTP surface. It counts as a miss, so a web scanner could find at most 26 of the 27 flaws.
• Blind SSRF needs a callback server. The sandbox blocks one, so the SSRF result is unconfirmed.
• For HSTS we graded only the vulnerable mode, which sends no header.
• These are small test apps with one flaw each, scanned with default settings by two tools. A real application is larger and has real dependencies. We make no claim about other scanners, paid or free.

Data files

• results.csv: one row per fixture, with the grade for every setup, alert and finding counts, and the alert and template names.
• counts.json: the count of found, partial and missed for each setup.
• run.sh: the reproduction script.
• README.txt and SHA256SUMS.
• web-security-fixtures on GitHub.