Some security teams are considering building AI pentesting in-house. They’re dreaming of plugging Claude Mythos, Claude Fable, GPT-5.5 and Gemini into homegrown agents and pointing them at their apps.
The first results look impressive. But dig a little deeper and the problems show up fast. A homegrown AI pentester:
- Finds vulnerabilities but doesn’t reduce risk.
- Can break production.
- Is expensive to run.
- Is unreliable and unpredictable.
- Requires world-class offensive security and AI expertise to build and maintain.
The reason is simple. A model can find vulnerabilities, but it can’t reduce risk on its own. That takes a full system around it, including validation, safety guardrails, application context, model benchmarking, remediation and retesting. Building that system takes skilled people, a real budget and months of work. Meanwhile, attackers are already using AI.
Anthropic learned this firsthand. Through Project Glasswing, it worked with partners to find vulnerabilities in widely used software. They found many, but cyber risk didn’t drop enough.
In Anthropic’s words: “It’s easier than ever to find vulnerabilities, but verifying, prioritizing, and fixing these findings remains challenging.” Fixes often took months.
That’s why, in its new Cyber Mission launched on October 8, Anthropic is pairing its models with on-site engineers, threat research, and partners like CrowdStrike, Palo Alto Networks, and Accenture for critical infrastructure. The approach goes well beyond giving defenders access to powerful AI.
The lesson for security teams is clear. Frontier models make finding vulnerabilities easier than ever. But the advantage doesn’t come from having the best model. It comes from turning that capability into measurable risk reduction, consistently and across every application.
Why Building Is Tempting
It’s easy to see the appeal. Frontier models are becoming more capable at finding vulnerabilities, and a skilled engineer can connect one to an application and potentially uncover real issues within days.
Attackers know this too. In a recent campaign against South Korean financial institutions, an attacker using an open-source pentest tool and general-purpose LLMs reportedly reached seven institutions in about a week.
The pressure to act is high, and many companies are already pushing every team to do more with AI in-house. Offensive security is a natural place to start.
But attackers only need one weakness to succeed. Defenders need to continuously find vulnerabilities, prove which ones are exploitable, and prioritize remediation across their applications.
The question is no longer just whether you can build an AI pentester. It’s whether you can operate one safely, reliably, and economically across your applications, week after week.
Building the Agent Is Just the Beginning
A model gets you findings. A system gets you results you can act on. Building that system takes skilled people, months of work and constant upkeep, while attackers are already using the same models with no reviews or approvals to slow them down.
If you’re considering building your own AI pentesting on Claude Mythos, GPT-5.5, Gemini or an open-weight model, ask these questions first.
Can you run it safely?
- Guardrails. AI agents testing live applications can delete data, lock accounts, or disrupt a service. You need hard limits on scope, blocks on destructive actions, and control over how intrusive each test is.
Can you run it reliably?
- Validation. Models can report issues that aren’t real. Findings need proof of exploitability before they reach developers.
- Context. Models don’t remember an application between runs. Finding business logic flaws and chained attacks requires an understanding of workflows, roles, APIs, and permissions that carries over from one test to the next.
- Continuous optimization. No single model is best at every task, and new ones are released constantly. You need to continuously evaluate models, agents, and tools, and improve how they work together.
- Remediation and retesting. Findings need actionable fix guidance, retests that confirm remediation worked, and reports auditors can use.
Can you run it economically?
- Cost. Frontier model tokens add up quickly when testing runs continuously across dozens or hundreds of applications.
- Maintenance. Models change, applications evolve, and engineers move on. Keeping the system effective requires ongoing investment and dedicated ownership.
Building isn’t impossible. But even with the strongest models available, turning findings into reduced risk is still the hard part.
Building your own means committing to running, evaluating, and improving an entire offensive security system over time.
What It Takes to Make AI Pentesting Work at Scale
At Novee Security, building offensive AI is all we do. Our team includes former nation-state offensive operators, security researchers and AI engineers. Our researchers have found zero-days in Gemini, Cursor, Microsoft and Google’s Python infrastructure, and that expertise is built into everything we make.
We approached AI pentesting as two problems. First, how do you build offensive AI that performs better and keeps improving? And second, how do you turn that capability into a security program enterprises can depend on?
1. We built the AI system, not just an agent
We built Novee from the ground up for offensive security, rather than relying on a general-purpose model with a few security tools attached.
We own and optimize the full offensive security AI stack. This includes a proprietary model we post-train ourselves, a purpose-built harness of skills and tools, and multi-model routing that sends each task to the best-performing model, whether ours or a frontier model.
Our specialized agents run on this stack. Together, they map each application, hunt for vulnerabilities, validate findings, and guide remediation.
But owning the stack isn’t enough. Attackers and AI keep changing, so the system has to keep improving too. That’s the job of the Novee Gym, our training and testing environment for offensive AI. We run the full system against thousands of real applications with known vulnerabilities and measure recall, precision, quality, speed, safety, and cost. Then we use the results to improve the models, routing, and agents. When a new model ships, we test it the same way and use it where it performs best.
And with persistent application intelligence, Novee builds an understanding of each customer’s workflows, permissions, APIs, and attack paths that carries over between assessments. Testing gets deeper and more targeted over time instead of starting from scratch.
The difference shows up in testing. In our latest benchmark, we compared Novee with Claude Opus running on a leading open-source harness, the closest thing to what a team would build in-house. Novee had:
- 2.1x higher recall, finding more of the real vulnerabilities
- 2x higher precision, with fewer false positives
- 6x more vulnerabilities found per $100
These results show why the architecture matters. Performance depends not just on the model, but on how the entire offensive AI system is built, evaluated, and optimized.
2. We built a security program, not just a testing tool
We designed Novee to turn offensive AI into a system enterprises can trust and run continuously.
Findings are independently validated. Novee uses independent validation agents to reproduce findings, with deterministic checks wherever possible, to confirm exploitability before issues reach security teams. Every validated finding comes with evidence, reducing false positives and manual triage.
Remediation is built into the process. Novee provides personalized, stack-specific fix guidance developers can act on, supports optional routing to AI coding agents, and retests to verify whether vulnerabilities have been addressed.
Safety operates at every layer. Novee uses model-level guardrails, hard controls outside the model that enforce scope, testing windows, and rate limits, and a separate Guardian agent that reviews actions before execution. But we don’t stop there. The Novee Gym continuously tests and optimizes the entire system for safety, including whether agents respect boundaries and avoid prohibited actions. We don’t rely on the agent to police itself.
Economics are predictable. Novee’s per-asset pricing makes continuous testing practical without runaway token costs or unpredictable model consumption.
When Building Makes Sense
Building can be the right choice, especially when you have a strong offensive security team, a narrow use case, or unique requirements that existing platforms can’t meet.
But building and buying aren’t mutually exclusive. Many teams use a dedicated platform for continuous testing across their applications, while their internal teams focus on specialized research, unique attack scenarios, and the risks specific to their business.
The same goes for findings. Novee’s Exploitability Validation can take findings from other sources, such as bug bounty reports, scanners, past pentests, or your own in-house AI tools, and confirm which ones are actually exploitable. Your team spends its time on real risk, wherever the findings come from.
Weighing Build v. Buy
AI has made finding vulnerabilities easy for everyone, including attackers. The teams that stay ahead won’t be the ones that built their own pentester. They’ll be the ones that started testing first.
If your team is considering building, start with the questions above. Then download our Build vs. Buy guide to compare what it takes to build AI pentesting in-house with what Novee delivers today.
Or book a demo to learn more.
