In a fireside chat hosted by Cyber Security Tribe, moderator Dorene Rettas sat down with Michael Rombout (Cyber Threat Intelligence, Southwest Airlines), Carrell Jackson (CISO, insightsoftware), and Ido Geffen (Co-Founder and CEO, Novee) to talk about continuous AI penetration testing: at what phase of adoption they’re at, and where they think it’s going.
Applications change constantly and evolve non-deterministically, attackers don’t wait for testing cycles to conclude, and the window between a vulnerability being introduced and exploited keeps shrinking — to the point where roughly 80% of public exploits now land before their CVEs are ever published. Point-in-time testing was built for a slower world. The conversation was about what replaces it, and how teams can tell exploitable risk apart from yet another list of findings.
Here are the highlights.
The attacker’s clock starts before the CVE
A security program shouldn’t anchor its response to when a CVE is published or when exploitation is first observed in the wild. It should run on the attacker’s timeline, which starts far earlier than a threat report.
- The existence of a critical CVE is information, not intelligence. What matters is whether something is actually being exploited, how, and whether that exploitation scales.
- Those are the questions a threat intel team exists to answer: not simply which CVEs crossed a severity threshold this week.
- Treat the CVE feed as the starting gun and, by definition, the defender is already behind.
“The clock doesn’t start when the CVE is published anymore. The attacker already started that clock before we knew there was a vulnerability.” — Michael Rombout
Continuous development demands continuous testing
A good snapshot still has real value, and Rombout specifically said not to write them off. The problem is cadence. Attackers don’t operate on an annual schedule, so an annual test can’t be the whole program.
Ido Geffen pointed out that applications change faster, and the kinds of applications are changing. With AI now embedded in software, applications behave non-deterministically — their outputs shift on their own. When development is continuous and behavior moves without a release, testing has to be continuous too.
Jackson drew the parallel to compliance. Point-in-time audits of SIEM logs used to be standard practice; today you can’t get certified without real-time, continuous monitoring. He expects the speed of adversaries to force the same shift in how testing is required and evaluated.
From a flood of findings to a focus on exploitability
Jackson named the problem every CISO recognizes: too much data, from too many tools. One source for internal assets, another for cloud, another for external exposure, another for CTEM. The job is turning that into a single, defensible read on risk — what he called a risk-composite metric.
- Start with exploitability. Then narrow: which of those issues are actually exposed publicly?
- Then, with agentic tooling, go a layer deeper — is the running component exposed in a way that’s genuinely reachable? Each filter pulls the volume closer to what truly matters.
- Geffen framed the same goal from the offensive side: find the entry points and attack chains most likely to produce an exploitable scenario, and trace where those paths connect to the crown jewels.
“We’re going to prioritize exploitability. But KEV information is overwhelming… so we ask, based on the tools and data I have, which of these are exposed publicly? And now, layering on agentic technologies, we can ask if the running component is exposed in a way that is exploitable. The closer you can get the microscopic down to the systems and processes running on the device, you can focus your teams on what truly matters and reduce risk.” — Carrell Jackson
What AI actually changes about offensive testing
Geffen marks the moment when AI became capable of transforming the economics of cybersecurity. As recently as two years ago, finding novel vulnerabilities or 0-days was closer to an art than a science — and most tools billed as scanners were really just matching fingerprints against known vulnerability classes.
- The turn came when advanced reasoning models arrived. Those models can chain steps together, reasoning from one weakness to the next. That capability is what kicked off the agentic era in offensive security.
- It also changed what’s testable. Older autonomous testing focused on infrastructure because infrastructure is relatively generic.
- Applications are specific to each business — their data, their workflows, their roles — and only reasoning-capable systems can probe that surface the way an attacker would.
“Up until two years ago, most of the tools used in automatic penetration testing were focused on infrastructure, because infrastructure is usually more generic. Applications are much more personalized for each organization… Advanced reasoning LLMs are capable of probing those types of attack surfaces in a way that couldn’t have been done before.” — Ido Geffen
The bottleneck has moved from finding to validating
More testing power creates a new problem: volume. Jackson’s instinct is to watch the signal-to-noise ratio closely — when there’s too much noise, the real signals get buried. The operational question is how to run continuous, agentic testing and still filter it down.
- Geffen agreed the constraint has shifted. Today’s frontier models surface plenty, but their false-positive rates are too high to accept at scale, and bug bounty programs are already drowning in AI-generated submissions.
- The hard part is no longer finding issues. It’s proving which ones are truly exploitable, and which actually touch the business and its threat model.
- Rombout’s test for any AI system is whether it can reason across findings and behave like an attacker; enumerate an environment, combine individually modest weaknesses, and demonstrate a real attack path. Done poorly, he warned, continuous testing can overwhelm incident response; its best use is eliminating theoretical risk and surfacing the paths that are real.
“I always think, signal vs. noise. When you have too much noise, are you missing signals? How do we operationalize continuous agentic penetration testing while continuing to help filter the noise?” — Carrell Jackson
“Can AI enumerate in an environment, combine individually modest weaknesses and demonstrate true attack paths in our environment? The value isn’t in the amount of findings it can create, but in something that’s actually exploitable.” — Michael Rombout
Autonomy has to be earned and governed
Every panelist drew a hard line between testing environments and production. Rombout described a “crawl, walk, run” progression: start with controlled analysis, expand autonomy only as it proves out, and keep anything that could affect physical safety — like OT environments — confined to a lab.
- Geffen catalogued the failure modes his team watches for: agents taking destructive action against a test environment, drifting out of scope, reaching toward third parties, or leaking data.
- Preventing those takes layered guardrails — soft ones that shape behavior through instructions, and hard ones that stop an agent from going where it shouldn’t. Just as important are reactive controls: kill switches and deep, auditable trace logs, so you can see exactly what an agent did and halt it.
- Jackson’s framing mirrored a traditional pentest. Rules of engagement exist for a reason, and an autonomous tester has to honor them — including not actually exploiting what it finds.
“I’m comfortable expanding the autonomy of AI when it’s controlled. There’s a big difference between allowing AI to enumerate in a testing environment and allowing it to do so in production. In my world, we start with controlled analysis and then move into validation.” — Michael Rombout
“Every AI company can have safety issues. It’s the basics of security to be able to detect, contain, and prevent in advance as much as we can. The mindset we have had for years when it comes to security for broader organizations, we need to take with AI.” — Ido Geffen
Specialized models and the economics of continuous testing
Cost has become part of the architecture conversation. Jackson noted that token-based consumption now makes AI spend something you track like cloud costs — and raised the practical question of whether redirecting a large annual pentest budget into continuous testing still satisfies certification requirements.
- Rombout added a caution: continuous validation has been building for years under banners like continuous exposure management, but testing without the ability to remediate quickly isn’t progress. Speed to find only counts if it’s matched by speed to fix.
- Geffen argued the economics increasingly favor specialization. Frontier models aren’t going away, but reinforcement learning on open-source models for a specific vertical — like offensive security — is becoming too capable and too affordable to ignore.
- Running general-purpose frontier models continuously is expensive and hard to forecast; purpose-built models change that math.
“In cyber it does make sense to have more specialized models.” — Ido Geffen
How to start building a continuous testing program
The panel closed on process, not technology.
- Geffen: Begin with what’s truly exploitable and a genuine threat to your organization. That’s the right entry point to a continuous program.
- Rombout: This is a question of program maturity. Every organization has more vulnerabilities than it can fix at once, so the real question is what an attacker could exploit today and whether it would materially affect the business. Bring external surface testing, attack surface management, penetration testing, and adversarial simulation together. Optimize to eliminate the attacker paths that matter most, not to generate more findings.
- Jackson: Translate all of it into business terms. Correlate what matters most into a risk registry so owners and the C-suite understand where the risk sits and what remediation will take. Articulating business risk through a security lens is what builds a program that lasts.
“Don’t optimize or program to find vulnerabilities. Optimize to eliminate attacker paths that matter most to your environment.” — Michael Rombout
Check out the full conversation for the complete discussion.
Want to see what continuous, validated testing would surface in your own environment? Book a demo.
