Novee’s Head of AI Dan Padnos and Founding AI Researcher Or Dancig go under the hood of Novee’s offensive AI, covering the environment it’s trained in, the benchmark built to measure it, and the controls that make attacker-grade reasoning safe to run continuously in production.
In our technical fireside chat, Inside the Novee Gym, Novee Head of AI Dan Padnos and founding AI researcher Or Dancig deliver a first-hand account of what it actually takes to train, evaluate, and safely operate an offensive AI system.
Offensive AI is a distinct engineering problem, and until recently it hadn’t had a real solution — or an honest way to measure one.
Watch the full conversation below.
Here are the highlights.
There was no honest benchmark for agentic offensive work. We built one
- Most public cyber benchmarks don’t reflect real offensive work, instead measuring code analysis or isolated exploitation rather than the multi-step reality of a live pentest.
- There was no dedicated benchmark for grey-box agentic pentesting at all, despite the fact that AI models were increasingly being used for exactly that use case. That’s why the team built PwnBench.
- Or Dancig walked through what PwnBench measures that existing benchmarks miss, and why an honest benchmark is the precondition for improving the system at all.
“Pre-existing benchmarks measure interesting capabilities of models, but they don’t measure a realistic penetration test. PWNBench does, in terms of the task, the target – which is web applications – the vulnerability types and the metrics that we measure.” – Dan Padnos
The system around the model is equally important
- To separate what comes from the model versus the system around it, the team ran 11 different models on the same harness and scored them head-to-head.
- Raw model capability is only one input. The harness — prompts, agent skills, tool orchestration — and the context the system carries between runs account for a large share of real-world performance. More details in the full PWNBench v1.0 research →
- This is the thesis behind owning both the model and the harness, and optimizing both continuously
“Evaluation isn’t simple. We see contradictions between multiple benchmarks. That’s why we needed to measure our agent on the end-to-end workflow; measuring it on specific phases, whether that’s discovery or exploitability, is good if you want to optimize those specific phases. But if you want to optimize the agent end-to-end, you must measure it on the specific overall task of pentesting.” – Or Dancig
Assumptions of existing benchmarks are disproven by a benchmark built for agentic greybox pentesting
- Code analysis does not necessarily equal strong pentesting capability, as evidenced by benchmarks that measure how well models can analyze code and find vulnerabilities (CyberGym) vs. those same models performance on PWNBench, which evaluates based on OWASP Top 10 vulnerability discovery and exploitation.
- Exploitability is not the entirety of the pentesting process; discovery is equally important.
- Recall and precision can diverge. Benchmarks that only measure recall favor more noisy reports.
“Whenever we build a system that handles a complicated task, reducing it to one number usually misses something important. What model is “better” depends on whether you want to measure cost, or precision, or cost. There’s at least one more important factor that comes into play, which is safety.” – Dan Padnos
On safety, and how it doesn’t have to come at the expense of performance
- An offensive system running continuously in production can cause real damage if it isn’t controlled. Novee frames that risk across five categories: damaging the target, going out of scope, hitting third parties, sabotaging its own test, and leaking data.
- Or shared what the team has found when measuring agents against those categories — including concrete examples of what an unchecked offensive system will do — and how Novee is built to prevent each one.
- Safety isn’t a disclaimer, it’s something you measure and engineer for
“The two things we did before we decided to see how our agents did on the safety issue: one, build an evaluator to see how they measure up against these five safety categories. We built an LLM-as-a-judge that ingests the traces and logs of the testing agents and determines if any of these five safety risks were breached. Two, run the Novee agent in a scenario where it actually performs zero safety violations, to establish a baseline.” – Or Dancig
“If you use a purpose-built agent like Novee, you get much more effort and a much more exhaustive pentest – as evidenced by a stack of 8000 tool calls vs. Claude Code’s 1000 – and you get it with zero safety violations.” – Dan Padnos
Why small, purpose-built models win on performance, cost, and safety
- Small, post-trained models aren’t a compromise. The team shared exploit rates comparing base small models against the same models fine-tuned through Novee’s training apparatus. Improvements as high as 2% to 51% in exploit rates were demonstrated. More details in the full deep-dive into fine-tuning SLMs for offensive security →
- A tighter, purpose-trained model is easier to constrain and reason about, which ties directly back to the safety work above.
“This is what you get out of the Novee Gym, a sophisticated laboratory that builds scenarios much like an actual pentest: Much better cost / quality tradeoff over trained frontier models, as well as better guarantees on safety and the ability to tweak and improve every detail on your harness and your model.” – Dan Padnos
The full conversation goes deeper on PwnBench, the 11-model run, the safety categories, and where offensive AI goes next.
The full conversation goes deeper on PwnBench, the 11-model run, the safety categories, and where offensive AI goes next. You can review the slides here.
Want to see what continuous, validated pentesting looks like in your environment? Book a demo. And to explore PWNBench for yourself, see it live on the Fireworks SII.
