In this conversation with Application Security Weekly, host Mike Shema talks with Ido Geffen of Novee and Abhi Patkar of Simpson Strong-Tie about moving from point-in-time pentesting to continuous, attacker-level validation.
Pentesting has always been a gate in Simpson Strong-Tie’s development process. The question Abhi set out to answer wasn’t whether to test, but whether a yearly report was ever really doing the job.
Watch the full conversation below.
Here are the highlights.
From an annual checkbox to continuous, developer-run testing
- Pentesting is a key gate in Simpson Strong-Tie’s software development process, but their prior vendor delivered one assessment a year and returned fairly basic reports. In practice it functioned as a compliance checkmark, not a real effort to surface exploitable risk.
- Since switching to Novee, the dynamic flipped. The software development team now engages with security findings far more, and prefers continuous testing to a scheduled, point-in-time assessment that only reflects a single moment.
- Because the testing is AI-driven, Simpson Strong-Tie was able to move past the tedious manual validation cycle. Once they let Novee learn their environments, the platform took on validation itself. It closes the loop between a finding and a confirmed, exploitable result without the team doing the legwork.
“I haven’t given you, meaning the platform, the context. My applications are running and the Novee platform is running. And without being handed context, it understands the whole environment, and shapes it with an asset intelligence model. That is impressive to me.” – Abhi Patkar
Chained exploits posed more risk than novel, individual findings
- Abhi wasn’t necessarily surprised by the individual findings. What stood out was how Novee analyzes what already exists and then tries to chain weaknesses together into entirely new exploits.
- In several areas, the platform chained lower-severity issues into a full session-chain exploit. These are the kinds of attack paths that only emerge when a system reasons across endpoints the way an attacker does.
- For Simpson Strong-Tie, that was the point: seeing how low-level issues, easy to wave off in isolation, could be extrapolated into something critical.
“I used to have to work very closely with our pentesting vendor to make them understand what the business logic was, what the various roles were, and so on and so forth. Findings were very basic or immature, and made me question if there was something beyond, if they could be chained. Novee was able to find that. Novee learns the application the way it is built and creates its own attack path. It created an attack path that went beyond the surface-level issues.” – Abhi Patkar
Prioritizing fixes means managing developer pride
- Once a high-severity exploit surfaces in someone’s code, developer reactions range from genuine surprise to immediate willingness to dig in. Prioritizing fixes is as much about managing that human response as it is about severity scores.
- The hard part is “denial of exploitability,” or the instinct to argue a finding isn’t really reachable. The only thing that resolves it is clear evidence.
- Novee delivers that evidence with every finding, so the conversation moves from debating whether something is exploitable to getting it fixed.
“The platform itself explains all the findings with detailed guidance and evidence, saying: ‘this is where the problem is reproduced.’ It’s all right there in the platform. Now, my developers are coming to me asking ‘Why do I need to run DAST when I have continuous testing with Novee?’” – Abhi Patkar
Earning the right to test in production
- Simpson Strong-Tie didn’t start in production. The first step was integrating Novee with their QA environment and proving it was trustworthy there.
- Once that integration held up, production was the natural next step — a progression earned through evidence, not a leap of faith.
- By then, Novee had caught so much upstream that production stayed essentially bug-free through the majority of deployments. The testing had already done its work before code shipped.
The findings curve
- In the early stages, Novee surfaced 10x to 15x more vulnerabilities than Simpson Strong-Tie’s prior solutions
- As findings were remediated, that volume dropped to a steady, manageable level. The spike was being caused by the backlog finally coming into view, then getting cleared.
“On the first go, Novee is doing a scan and giving you a lot of output, but once you tell the platform, in just a few clicks, what is permissible and what isn’t, the platform continuously learns and produces more signal. That said, it is still difficult to understand the context of business logic. Novee was able to understand that there were situations where an admin role’s privileges were being carried down to a low-level role; it was impressive that Novee was able to discover issues at that granular level.” – Abhi Patkar
Developers in the pipeline, AppSec on oversight
- Developers are the primary users. They’ve wired Novee into their CI/CD pipeline, so testing runs as part of the deployment process rather than as a separate event.
- AppSec uses Novee for due diligence and auditing. They track who’s doing what, and follow up when remediation goes undone. Previously, that oversight ran through a manual tool.
- AppSec can see every report and step in when needed, but stays hands-off from running the testing itself. Security coverage scales without becoming another team’s full-time job.
“In the name of guiding remediation, we are trying to work as closely as possible with – and make the platform as seamless as possible for – developers. For example, we know developers prefer to work with API. We can also integrate directly with the CI/CD processes and communicate directly with coding assistants via our Agentic Fix capability. That means, when there’s fresh code, we can test it and immediately send to their coding assistants of choice what was found, how it could be exploited, and how to fix it.” – Ido Geffen
Trusting an offensive agent not to go rogue
- Simpson Strong-Tie trusts Novee because they understand how it works, know how to audit it, and take advantage of the system’s built-in guardrails.
- In practice, they’ve watched the system stay where it’s supposed to. They haven’t seen their data exploited by Novee, and the testing agents haven’t taken malicious action. A few speed-throttling hiccups turned out to be configuration-side issues, not the agent misbehaving.
- In Simpson Strong-Tie’s view, pointing fingers at “the AI” skips the harder, more important question of your own internal governance and review policy.
“You need to have guiding principles of guardrails and a process to go with them. Once you see the output, once you see how quickly the tool learns, and once you see how safely it operates in a testing environment, you build that trust.” – Abhi Patkar
How Novee engineers agent safety
- Ido walked through the layered work behind safe autonomous testing. Soft guardrails shape agent behavior through instructions; hard guardrails constrain where an agent is allowed to go at all.
- Those hard guardrails cut traffic the moment anything goes out of scope — if an agent exceeds rate limits or reaches toward an unauthorized endpoint, it’s stopped.
- On top of that, Novee runs a “guardian”: a separate agent that independently validates every action a testing agent plans to take before it happens. Safety isn’t a disclaimer bolted on afterward — it’s engineered into how the system operates continuously in production.
“Even a year ago, before Hugging Face and ‘rogue agents,’ our internal research lab – the Novee Gym – saw AI agents trying to get out of testing environments. It’s a real concern in the end, that these types of agents can go rogue, so we came up with layers of combined protection.” – Ido Geffen
“Soft guardrails reduce dangerous actions, but we understand it’s not enough. Hard guardrails tell agents, ‘do not touch this endpoint, do not get above this latency, do not pursue testing beyond an error message.’ All of these are hardcoded like a firewall.” – Ido Geffen
Challenges and next steps
- Ido: The bottleneck has moved. Discovery used to be the hard part; now it’s validation. Autonomous testing covers more ground in the first place than manual, and validating all those findings becomes the KPI to chase.
- Abhi: The toughest work is complex fixes — bugs Novee found that could be extrapolated into different applications in different ways, where the underlying architecture made remediation genuinely difficult. Novee now helps here too, generating a software architecture layer diagram and a risk model that make those cross-cutting fixes tractable.
“It is agentic, it really is at that frontier where things just work smoothly.” – Abhi Patkar
Check out the full conversation for more.
Want to see what continuous, validated pentesting looks like in your own environment? Book a demo.
