NoveeGym-RL: A Post-Training Pipeline for Offensive Security

Presenting the architecture and design choices underlying Novee’s LLM post-training infrastructure for agentic pentesting.

Post-training is the lever

The best offensive security AI will not be a clever prompt on top of a general-purpose model, but rather a purpose-built, vertically integrated system, where both the harness and the model are specialized for the unique requirements of offensive security. Novee’s goal is to build the best pentesting agents, using a continuous-improvement loop that we call the Novee Gym. This post describes the part of the loop that improves our proprietary models through reinforcement learning: NoveeGym-RL.

Finding and exploiting a vulnerability in a penetration testing scenario is a long-horizon, tool-using task: probe a running system, form a hypothesis, write an exploit, watch it fail, revise. A base model has fragments of this capability scattered through its weights, but it hasn’t been specifically optimized for doing the full loop well. Post-training is how you reinforce the behaviors that close the loop; how you take a model that can reason about a stored-XSS bug, and turn it into an agent that reliably lands one.

Post-training takes two forms, typically applied in sequence:

Supervised fine-tuning (SFT) installs format and baseline competence by imitating successful episodes token by token.

Reinforcement learning (RL) improves the policy beyond what imitation alone can reach, learning from experience: executing full rollouts, judging their outcomes, and reinforcing successes over failures. 

Pentesting is well-suited to RL because success is mechanically verifiable: an exploit either executes or it doesn’t. Clean, cheap, un-gameable verification can be non-trivial to implement, but once achieved it puts us in the regime of RLVR (reinforcement learning from verified rewards), where progress is unbounded by human supervision or the signal from a teacher model. Over the past 18 months, RLVR has become the dominant paradigm for post-training agents on reasoning-heavy tasks like coding and math, and it is the centerpiece of our pipeline.

A key feature of our approach is that we train the model inside the harness it will be deployed in — the same skills, tools, and prompts — so it specializes to the exact agent it will drive. And the loop feeds itself: an agent trained this way finds fresh zero-days in the wild, which become training data for the next round.

Why we built our own

The RLVR boom that followed OpenAI’s o1 and DeepSeek-R1 was built around single-turn, verifiable tasks: pose a math problem, sample a reasoning chain leading to an answer, check the answer, reward it. Most open RL frameworks inherited that shape and they’re excellent at it. They are a poor fit, though, for what we needed: a black-box agent taking hundreds of tool-using steps against a live, stateful web app, driving real browsers, graded by a deterministic exploit validator that also needs access to the same live resources.

The key design principle was decoupling. We keep the moving parts — model inference, model training, environment, harness, and reward — behind clean interfaces, joined by as little glue code as possible, so any one can be tweaked or replaced without disturbing the others. Early RL frameworks did the opposite: they baked the harness into the rollout orchestration, with opinionated decisions about tools and context management, and they made rigid assumptions about what an environment even is and how its lifecycle is managed.¹ Working at the bleeding edge of open models adds its own challenge: new models often need custom kernels and custom handling of reasoning and tool formats, and it takes time for those to get merged into popular frameworks.

We set a high bar for NoveeGym-RL; efficient, reliable, and hackable on a short timeline. Achieving all three would not have been feasible for a small team a year ago. Two things made it possible:

First, open-source foundations we could stand on and reshape, instead of starting from a blank file: verl for the distributed trainer, rLLM for the agent-RL loop, vLLM for efficient inference serving, and Axolotl for end-to-end SFT (which is outside the scope of this post). Second, AI coding assistants, which let us experiment and refactor at a velocity that simply wasn’t available before.

Anatomy of a rollout

Each rollout is one agent’s full session attempting to complete one task. The RL loop is fed by many concurrent rollouts orchestrated in parallel. To handle the lifecycle of a single rollout we implemented rLLM’s AgentFlow protocol, plugging our custom provisioning, sandboxed execution, and evaluation into it. This is described in the following (simplified) code snippet:

class NoveeAgentFlow:  # implements rLLM AgentFlow protocol
    def run(self, task: Task, config: AgentConfig) -> Episode:
        # (1) acquire external resources
        target = webapp_pool.acquire_app_instance(task.metadata["app_id"])
        browser = browser_pool.acquire_browser()

        # (2) rollout
        prompt = task.instruction.format(target_url=target.url, browser_cdp_url=browser.cdp_url)
        model_gateway_url, model_name = config.base_url, config.model
        rollout_config = {
            "sandbox": {
                "type": "docker",
                "image": "novee-abc123",
                "allow_urls": [target.url, browser.cdp_url, model_gateway_url],
            },
            "model_base_url": model_gateway_url,
            "model_name": model_name,
            "harness": "novee-custom-harness-v0.1",
            "max_steps": config.metadata["max_steps"],
        }
        result = run_rollout(prompt, rollout_config)

        # (3) validate
        verdict = validate(result, target, task.metadata["validation"])

        # (4) free external resources
        browser_pool.stop(browser)
        webapp_pool.stop(target)

        # (5) return validation verdict
        return Episode(..., artifacts={"verdict": verdict})

This is not a precise reproduction of our implementation (e.g. we omit error handling, logging, and other nuances) but it captures the essence of the flow.

The objects webapp_pool and browser_pool are global utilities that provide interfaces for managing fleets of target apps and browsers respectively; we describe them in more detail below. Together with the sandbox container itself, these resources can be viewed as the environment the agent operates in. The environments need to be completely isolated across parallel rollouts, since agents mutate the environment state as they work — storing payloads, editing records, logging in — so two rollouts sharing an app or a sandbox would corrupt each other’s evidence.

The heart of NoveeAgentFlow is the rollout execution. In the snippet above, run_rollout is an abstraction that executes a rollout to completion (or until hitting a step budget) and returns a structured result. The underlying harness and sandbox providers can be swapped for different implementations. The key design choice enabling this flexibility is the model gateway, which acts as a proxy between the harness and the inference servers, enabling us to collect traces for training without touching anything internal to the harness or sandbox. ²We describe its function in more detail below.

Our preferred approach is to deploy each agent in its own disposable Docker container, hosted either locally or on a remote node pool (via k8s). Each rollout needs network connectivity to its app instance, browser instance, and the model gateway. We found that the cost of managed sandbox providers wasn’t justified when we already run our own cloud compute. In practice, a multi-GPU box has plenty of spare CPU: with tuned Docker and machine settings we comfortably run 500–1,000 concurrent rollouts on a single 8×B200 node.

The result returned from run_rollout is passed to a validation function, which gives a verdict on the correctness of the result. The verdict is then read and converted to a numerical reward by a lightweight NoveeEvaluator implementing rLLM’s Evaluator protocol. The custom NoveeEvaluator and NoveeAgentFlow are glued together by a customized workflow that wraps rLLM’s UnifiedTrainer, supporting different training backends (verl and tinker).

In the following sections we provide additional details on the key dependencies of NoveeAgentFlow: the instance and browser pools, the validation, and the model gateway. Then, we proceed to close the RL loop by describing async inference and training.

Provisioning application instances and browsers

Each app instance is a live deployment of a real open-source web application, not a simulator or a mock, running on its own host and reached over the network the way any client would. Its state persists through the rollout as the agent probes and exploits. We orchestrate these deployments using our own custom infrastructure over Docker Compose and a fleet of EC2 instances. App images (as well as metadata linking apps to tasks) live in a dedicated repository. At peak the pool runs thousands of concurrent app instances for days at a time.

Spinning up a real app, including database, services, seed data, authenticated sessions, takes time, as much as a few minutes in certain cases. This is too long to execute inline with GPUs idling while we wait for the app to reach ready state. To maximize GPU utilization, we overlap app startup with rollouts by provisioning app instances ahead of demand. The webapp pool reads some number of training rows ahead of rollout execution and fires off async instance launch requests to serve future rollouts. This “lookahead demand” gets served in the background while other rollouts execute, creating a pre-warmed pool of app instances. When we execute a rollout, acquire_app_instance() first looks for an available pre-warmed instance before launching a new one.

For a deeper view on how we collect the web applications themselves, see our blog.

We adopt a similar shape for browser instances. A centralized server starts and stops isolated browser sessions on demand, which our agent can drive through CDP (Chrome DevTools Protocol). Since browser start time is <1s, there’s no need to pre-warm browsers ahead of demand, so we drop this from the browser pool implementation.

Verifiable rewards

How do we assign a reward to an agent rollout? 

The easy answer is to have the agent write up what it found and use another LLM to grade the writeup. That’s a biased and unreliable method. Fundamentally, it rewards persuasive prose over working exploits, and an LLM judge is exactly the kind of soft signal a policy learns to game. So we don’t grade claims, we grade execution. A deterministic validator takes the agent’s exploit, runs it against the app, and checks whether the attack actually fired.

This is easier said than done. The skeleton stays the same, but different vulnerability classes, and even different app architectures, require vastly different implementations of deterministic validation; each needs a different encoding of what precisely counts as “validation fired.” We built an extensible framework where adding new cases is a small plug-in, not a rewrite. In the example we share below, we focus on one class of vulnerabilities (XSS) across any webapp architecture.

The model gateway

Between the agent and the GPUs that serve the model sits a model gateway — a fork of rLLM’s, extended for observability, scalability, and TITO (tokens-in, tokens-out). The gateway provides an ordinary Chat Completions API, same as a test-time rollout, so the agent doesn’t know it’s running in an RL loop. In parallel, the gateway holds a side channel to record that traffic and make it usable for training.

A key requirement for efficiency and accuracy is achieving tokens-in, tokens-out (TITO) inference. A normal Completions API takes text and returns text, but LLMs operate on token sequences. This requires a tokenization roundtrip that is sometimes lossy, leading to silent corruption of the gradients and/or bloated training batches. We’ll dive deeper into this point in a future blog post.

For inference we use vLLM. The current policy is served with vLLM on its own GPU pool. It handles the high-concurrency generation load from all the in-flight rollouts. New weights coming from the trainer are pushed in without tearing the server down. The gateway implements “sticky routing”, where we prefer to route each inference request to the same GPU that ran the preceding steps of the same rollout. Since KV cache is GPU-local, sticky routing maximizes cache hits and improves throughput considerably.

Async RL

Modern RL frameworks run fully asynchronous RL, where rollout generation and training happen simultaneously on two separate GPU pools, instead of a synchronous pipeline where they execute in lockstep. Async RL reduces idleness and improves throughput. We reap the async benefits by leveraging rLLM and verl’s async RL implementation, which links the GPU pools by a buffer. The rollout worker runs agent flows and drops finished, scored episodes into the buffer; the trainer worker pulls from the buffer as episodes land and runs policy updates; updated weights are transferred back to the inference server asynchronously. Neither side waits for the other, and that decoupling is the throughput win.

In GRPO and similar group-based methods, the trainer consumes groups with nonzero variance in reward value, to support advantage calculation. To collect a training batch, we first drop rollouts that ended in an error (such as a mid-rollout infra failure). Then, to apply the nonzero-advantage constraint we drop groups where all the valid outcomes have the same reward value (all pass or all fail). Remaining groups are eligible for training.

The cost of async RL is built in: episodes are generated while the policy keeps training, so the data always comes from a slightly stale policy. The loop is off-policy by nature. Taming that drives most of the complexity of async RL; draining in-flight requests at each weight sync, double-buffering served weights, versioning every call, and triggering weight syncs to bound staleness so the gradient stays honest. Exactly how far to push staleness is still an open research question for us, but capping the lag at about four steps already gives stable training with clear reward gains, while almost doubling the throughput. It’s a deep enough topic to be its own post; for the time being we note that the orchestration layer keeps both pools saturated while holding staleness inside a safe envelope.

Training: the objective, masking, clipping and batching

The objective. Our workhorse is GRPO, with the common adaptations, all supported through verl; we’ve also implemented DPPO. Advantages are group-relative — several rollouts per task, scored against each other — which is what makes the rollout group the natural unit for both training and the observability tooling below.

Masking. We mask everything the model didn’t generate, so gradients flow only through the agent’s own actions, never the environment’s tokens (e.g. tool output). Getting the mask exactly right is essential because a single misaligned token boundary would train the model on text it never chose. (This is also where TITO pays off: the token ids we mask on are the same ones the policy generated.)

Clipping. Because the loop is off-policy, the trajectories were generated by a policy that differs from the one being updated. We correct for that with an importance-sampling ratio, and we clip it — a PPO-style trust region — so a single stale, high-ratio sample can’t blow up the update.

Batching. A single episode’s many calls are packed into a compact set of training rows before they reach the trainer, so a batch stays dense rather than padded out to the longest trajectory — the difference between a GPU that’s working and one that’s mostly waiting. While straightforward for classical Transformer models, packing becomes trickier for hybrid models combining linear attention or SSM layers with classical attention. While hybrid models are increasingly popular, framework support for various features is often lagging. Specifically, at the time of implementation, none of the relevant frameworks supported sequence packing for Qwen-3.5 family models (starring in the example below), so we had to roll our own.

Observability for async RL

Async RL is hard to debug precisely because nothing happens in lockstep: at any instant, hundreds of rollouts are mid-flight, some episodes are queued in the buffer, some are being packed, and the trainer is a few weight-versions ahead of the policy that generated what it’s about to consume. Scalar dashboards tell you that something went wrong, rarely where. So we lean on three layers: heavily-extended W&B metrics, persisted detailed logs and traces, and — most importantly — a custom web UI that gives us a ringside seat on rollouts, buffers, and weight versions.

The rollout group is the first-class object. For any group you can see its members’ current state — in progress (and at which step), completed (with its reward and the referee’s verdict), or queued for training — and inspect the full trajectory of any single rollout: every action, tool call, page, and the exact tokens and logprobs. That makes the specific pathologies of async RL visible: staleness, reward-distribution skew within a group, and rollouts that stall or die in provisioning rather than in the agent.

Example: RL for hunting XSS

The machinery above is general. To make it concrete, we present end-to-end training on one task: teaching a small model to find and exploit cross-site scripting (XSS). 

In XSS, attacker-controlled input is reflected or stored by an app and later executed as script in a victim’s browser. It’s a good educational example: common, high-impact and cleanly verifiable (using purpose-built deterministic validators).

For this example we focus on the exploit phase, meaning our agent rollouts start from a hypothesis regarding where an XSS vulnerability might be found. The level of specificity of the hypothesis may vary (e.g. pinpointing the precise sink vs broadly referring to a feature in the app) and we cover different types of XSS (reflected XSS, stored XSS and DOM XSS). The agent is given the hypothesis and required to provide a working exploit conforming to our validation requirements. When an agent concludes it has a working exploit, it can submit the exploit code, terminating the rollout. A deterministic validator then inspects the submitted exploit code, checks it with various anti-cheating mechanisms and runs it against the live app to confirm it triggers the vulnerability correctly.

Below we report the results of end-to-end training of a small open-weights model, Qwen-3.5 4B, which is cheap and fast to train and run. The model’s initial performance on the held-out evaluation set was very low, 3% pass@3. We started with a round of SFT on a small, 300-example corpus of successful trajectories, to install basic skills and formatting required in this task, as well as get rid of unwanted biases of the original model such as unnecessarily long reasoning. This produces a warm-start checkpoint which we then feed into our RL training pipeline. For the RL phase we used a separate training split with 1300 unique tasks, calibrated in difficulty to reduce the number of zero-advantage groups (too difficult/easy tasks resulting in all fails/passes respectively). We used a batch size of 16 groups with 8 rollouts per group. For both SFT and RL we used LoRA with rank 64 and alpha 128.

The plots below tell the story. Training reward climbs as RL proceeds, meaning the agent lands more exploits per rollout over time. More importantly, performance on a held-out evaluation set rises alongside training reward, showing the gains are real capability rather than memorization of the training examples.

Conclusion

The XSS run is a single worked example, but the point is the pipeline behind it: a small open-weights model, post-trained end-to-end on verified rewards, measurably improves at a real offensive-security task. The machinery that made it possible (provisioning live environments, a gateway that captures clean training signal, fully-async RL, and deterministic validation) is deliberately decoupled from XSS and from any single model. Any task we can pose as a black-box agent against a live environment with a mechanical check of success can be trained the same way. That is the foundation NoveeGym-RL is built to keep extending.

In future blog posts, we will dive deeper into specific technical challenges we had to tackle to build NoveeGym-RL, and share how we use it to train increasingly capable models to power Novee’s pentesting agents.

1. The research and open-source community is now converging on this black-box-agent framing. For example: NVIDIA’s Polar, Prime Intellect’s verifiers-v1 and Fireworks’ RL cookbook all point toward decoupling the agent, environment and the RL machinery from each other. ↩︎
2. Similar to Prime Intellect’s InterceptionServer, recently introduced to verifiers-v1, as well as the “shim” provided in Fireworks’ RL cookbook. ↩︎