PWNBench-v0.1: Evaluating frontier models for web application pentesting

PWNBench-v0.1 summary results. End-to-end penetration testing performance versus cost for different models using a fixed, thin harness. The vertical axis shows F_0.5, which combines precision and recall with extra weight on precision. The horizontal axis reflects cost on a reversed log axis (expensive left, cheap right). Each curve corresponds to one model where test-time compute is increased by adding parallel runs and merging their outputs. Reasoning efforts indicated in brackets in the legend. Values reflect all finding severity levels combined with equal weight.

Novee’s approach to building the world’s best pentesting AI depends on evaluating every layer of the agentic stack — the model and the harness. That evaluation problem has gotten harder. Offensive capabilities in frontier LLMs have accelerated quickly, and every model release now claims a jump on some mix of general-purpose evals and cyber-specific tests.

We have found those public signals useful, but incomplete. They are also biased relative to the work we actually care about: end-to-end pentesting of live web applications. Our internal suite is built for that workload, and it is still growing. Today we are publishing results from an early slice of it, PWNBench-v0.1 (Pentesting Webapps — Novee Benchmark), covering 11 recent models from 8 labs — closed frontier APIs and open weights.

We will keep expanding and updating the benchmark, and we are considering open-sourcing some form of it in the future.

Existing benchmarks fall short

The job we want to measure is agentic greybox pentesting of web applications. The tester gets ordinary account credentials and limited supporting material — a user manual, API docs — and has to find real issues in a running app. That is the bread and butter of offensive work on web apps: it matches authenticated attack scenarios, and it uses testing time efficiently.

There is no shortage of prior work. Academic and open-source efforts include ExploitGym, CyberGym, and ExploitBench. Labs and security companies have published some results of their own. For our purposes, each of these misses the mark for some combination of:

• measuring only exploitation, not the full discover-exploit-report workflow
• not targeting web applications
• relying entirely on recall of CVEs the models have almost certainly seen in (pre-)training
• assuming source-code access (whitebox)
• exclusively targeting planted bugs in vibecoded applications

PWNBench is built to close those gaps. Each instance gives the agent typical greybox information and asks it for a set of validated security issues. We do not hint at the vulnerabilities.

The applications are forks of large, well-maintained open-source projects — software development platforms, CRMs, observability tools, personal productivity apps, ERPs, and similar systems.

We score results by matching reported issues against an extensive ground-truth database built by human experts. The database covers zero-days and some CVEs, and leans heavily toward zero-days. On top of pre-existing vulnerabilities, we make targeted code changes that introduce challenging vulnerabilities, inspired by real-world examples we encounter in production. We validate that every injected vulnerability can be discovered and exploited in practice. If an agent’s finding can’t be matched confidently to a known label, we run the same human-in-the-loop annotation process and extend the database. For v0.1 the benchmark includes more than 400 labels, covering all OWASP Top 10 risks.

Finally, PWNBench also scores precision, not only recall, and allows us to break results out by severity (info/low/medium/high/critical).

The models under evaluation

We ran 11 models from 8 labs:

We used the same harness for all models. It is not Novee’s production stack — but a significantly simplified, thinner version, with one model per run and without multi-model routing and other production-only machinery. Absolute scores should not be read as actual pentest performance. We find that this lean setup still gives a fair, realistic comparison of the relative strengths and weaknesses of different models.

We treat test-time compute as a first-class variable, with two knobs:

1. Reasoning effort — controlled by API parameters.
2. Multiple runs — run the same configuration k times and merge the findings. Each point on a recall curve is one value of k. Cost scales roughly linearly with k; recall usually rises sub-linearly because different runs report partly-overlapping sets of findings.

Results

The plot at the top of this blog post shows F_0.5 scores, which provide a single figure of merit combining precision and recall with extra weight on precision — a cleaner report achieves higher F_0.5 than a noisier one with similar coverage.¹ In this high-level summary, Grok 4.5/4.6 and DeepSeek-V4-Flash-0731 dominate the efficient frontier, with other models offering comparable quality but at significantly higher costs. The remainder of this section dives deeper into the results, disentangling recall and precision and analyzing finding severity.

PWNBench-v0.1 eval results. Top: recall vs. cost. Bottom: precision at each model’s highest plotted k. Left: all findings. Right: medium or higher severity findings only. Markers are company logos. The k= label annotates the top point of each curve. Recall and precision values are macro-averages across applications. Cost is total API spend. Precision error bars are the standard error of the mean across apps.

Recall always costs money, but the exchange rate varies a lot. Results show that recall consistently scales sub-linearly (roughly logarithmically) with cost within a model family. Comparing across models, we see dramatic difference in cost efficiency.

• When looking at all severity levels, the pareto frontier is spanned (low-end to high-end) by DeepSeek-V4-Flash, Grok 4.5 and Kimi K3. Claude Opus 5 with max reasoning effort is significantly more expensive, which does buy higher recall but at a steep price (51%/$1400 for Opus 5 at k=3 vs. 42%/$209 for Kimi K3 at k=3).
• Considering only medium or higher severity issues doesn’t alter the conclusions significantly, with the exception of Grok 4.6 which moves to the high-end of the efficient frontier.
• Claude Opus 4.6/4.8, GPT-5.6 and GLM5.2 span the 20%-30% recall range at a significantly higher cost than the pareto frontier (roughly $200-$600 depending on the model vs <$60 for DeepSeek-V4-Flash).
• Nemotron 3 Ultra and Muse-Glimmer-30B are at the bottom of the recall ranking (<15%), with a significant price difference between them (Glimmer ~10x cheaper than Nemotron).

Precision is a separate ranking. High recall does not imply a clean report. On all severities and also when filtering for medium+, Grok 4.6 and Opus 4.8 consistently stand out in terms of precision, with scores in the high-70s to low-80s. Despite its excellent recall and efficiency, Kimi K3 lands in the bottom third of the precision ranking, with scores around 50%-60%. Many models enjoy a relative increase in precision when narrowing the scope to medium or higher severity issues, including Opus 4.6, Opus 4.8, DeepSeek-V4-Flash-0731 and Kimi K3. Other models either keep their score or deteriorate to some extent.

More reasoning pays off for frontier models. For example, GPT-5.6 Sol (max) beats Sol (medium) comfortably on both recall and precision, for a similar cost. Opus 4.6 (max) and Opus 4.6 (high) show the same pattern. Note that results are not as consistent in open weights models. For example, GLM5.2 gave very close results in terms of both cost and eval metrics, regardless of whether thinking was turned on or not (we kept only the no-thinking results in the plot for readability).

Takeaways

There is no single “best model” on this benchmark. The efficient frontier is held by Grok 4.6, Kimi K3, Grok 4.5 and DeepSeek-V4-Flash-0731, roughly in descending order of cost and recall. For an overall effective pentest — high-severity issues, few false positives, reasonable spend — Grok 4.6 is likely the model to reach for first. However, if the objective is maximum coverage and budget is a secondary concern, then Opus 5 is ahead. These estimates come with a considerable measure of app-to-app variance, as the precision errorbars demonstrate. We therefore recommend treating these conclusions as general guidance rather than a definitive ranking.

Two methodological points matter as much as the ranking. First, precision and severity change the story. A recall-only, all-severity score flatters models that are eager to report. Filtering to medium+ and looking at false-positive rate is closer to how a human pentester would judge the output. Second, test-time compute is a real lever. Increasing reasoning effort and/or number of parallel runs moves a system along the recall–cost curve as much as switching a model. Comparing a single default sample from each API is not a fair or useful measurement of pentesting capability.

PWNBench-v0.1 is an early cut: a simplified harness and a ground-truth database we are still extending. We are sharing it because the spread across current frontier models is already large enough to be useful, and because we want the public conversation about “cyber evals” to include the workload we actually run. We will publish updates as the suite grows and we learn more.

F_0.5 = (1 + β²) PR / (β²P + R) with β = 0.5, i.e. F_0.5 = 1.25 PR / (0.25P + R), where P is precision and R is recall. We compute this per application and then take an unweighted mean across apps.