Bittersweet Lessons Learned Training LLMs on Long-Horizon Pentesting Tasks

Nobody tells you that the hardest part of teaching a model to hack isn’t the hacking.

When we started training Novee’s proprietary offensive security LLM, we knew it would be a challenge, but we expected the difficulty to live in the hacking itself. Open-weight models were getting better every day, open RL research was advancing rapidly, and we had a clear plan for teaching one to find real vulnerabilities in live systems.

The model did learn to hack, and did so well. But the deepest challenges came from an endless list of little things that quietly go wrong when you train a large model on long-horizon tasks. Complex systems have many moving parts, and getting them to work in concert took real time and effort. Along the way we ran into a handful of problems that enabled us to understand the nuances of our system.

The RL-for-LLMs landscape is full of countless papers, framework docs, and blog posts. We wanted to capture what most of them skip: the strange, specific edge cases that only bite once you are deep in a real run. We’re publishing these notes alongside a broader blog that walks through our high-level system architecture; think of this collection as the close-up that adds color on top of that wider treatment. 

None of what follows is groundbreaking, and it might look obvious in hindsight, but it did not look obvious while it was happening (with a run silently diverging and the GPU usage meter ticking). Each lesson cost us real time and real money.

So here they are: The notes we wish someone had slipped us before we started.

1. Prefix breaks quietly bloat your batch

Most of our infrastructure work comes back to one number: how fast training runs. As it turns out, there is a subtle but important detail that can affect training speed, and it goes back to how a multi-step agent conversation flows back and forth between the model and the harness.

Consider a simple agent loop. On each iteration, given a list of messages (the system prompt, the conversation history so far, and the available tools), the loop does the following:

1. Generate a completion from the language model.
2. Extract the tools the model is invoking from the completion, i.e. the actions.
3. Run the actions in the environment to obtain observations.
4. Wrap each observation as a new message and append it to the list for the next iteration.

The inference engine (vLLM, in our case) sits in the middle of this loop, and the classical implementation treats it as a messages-in, messages-out interface. Since language models operate on sequences of tokens rather than message lists, the inference engine needs to handle the two-way conversion internally: it renders the messages (using chat templates), tokenizes the resulting string, invokes the model, decodes the model’s output, and returns a parsed message (using tool and reasoning parsers).

An agent rollout consists of iterating through this loop, which produces pairs of token sequences (prompt_ids_k, completion_ids_k), where at each step k the new prompt consists of all the past conversation, together with the new observation from the environment. Each such pair is a perfectly good training example we can learn from. But treating them as separate training examples along the batch dimension is inefficient, because we recompute the same conversation history in consecutive steps. Instead, we can pack them into a single sequence and mask out the observations coming from the environment from the loss computation, so that we only optimize the actual output produced by the model.

This simple trick speeds up training steps by a factor of the number of steps in the rollout, which is easily 20-100 for long-horizon tasks like pentesting. But it hinges on a subtle assumption: each prompt_ids_k must be a precise token-level prefix of prompt_ids_{k+1}. In practice, this assumption doesn’t always hold. Such cases are called prefix breaks, and they result in greatly inflated batch sizes.

Prefix breaks carry other efficiency costs on top of the inflated batch size. A prefix break also directly implies a cache miss in the inference engine’s KV cache, since the old sequence is no longer a prefix of the new one. That means lower inference throughput, which slows down rollout accumulation and drags down overall training throughput.

Several things can cause prefix breaks, and they all ultimately stem from the decode-then-re-tokenize round-trip. One rich source of these breaks is the chat template. The chat template is the Jinja script that renders a structured conversation into the single flat string the model actually reads. The template decides exactly how each turn, and each tool call, is laid out as text, and that is the same text our tool parser later has to read back to recover what the model asked for. It is a treacherous little corner of the ecosystem: every model ships its own Jinja, they are under-tested, and the community runs on a steady supply of subtle rendering bugs and competing fixes. Booleans, for one: our tool calls carry JSON like {“headless”: true}; the model samples the token true, but the template re-renders it through Jinja’s | string filter as Python’s True; similarly, false becomes False, and null becomes None. The semantics are the same, but the tokens are different, so we get a prefix break.

Even a perfect template does not save you, because the tokenization itself is not stable across a round-trip. Most LLMs tokenize with Byte-pair encoding (BPE), which is greedy and canonical, meaning it always makes the same merges. The model, though, samples tokens one at a time and can walk a different, equally valid token path that decodes to the same characters. So decoding and re-encoding can quietly hand back different tokens for identical text.

The community has been working through this set of problems for a while now. The emerging cure is a token-in-token-out (TITO) approach, where the inference engine consumes tokens and returns them directly. This places a bigger burden on the harness (or the client): turning tokens into actions, turning observations into tokens, and carrying the token history forward across steps. Getting this right requires careful implementation, but the upside is significant. For example, in our implementation, eliminating prefix breaks resulted in a roughly 5.4x decrease in packed batch size. 

We were glad to see Prime Intellect’s recent release of renderers, and we recommend their write-up as an introduction to TITO (HuggingFace’s TITO writeup is excellent too). Renderers provides hand-written utilities per model that keep messages and tokens in sync across a conversation without round-trips. 

We solved this for our own models, but we like the initiative, and are considering adopting renderers in our codebase.

2. Silence is the most expensive kind of failure

A bug that crashes is a gift. It tells you where to look.

The bugs that hurt us most did not crash. A casual glance at the reward or utilization dashboards won’t reveal any issues. The run will go on, and silently degrade, training a slightly worse model while everything looks fine.

The first sign of our most elusive bug was in the gradient norms. Every so often, with no visible pattern, a step would post an unusually large or outright NaN grad norm, and then the run would carry on as if nothing had happened. Most steps were completely fine. Chasing that sporadic NaN led us to a non-deterministic fault buried in a single kernel on the linear-attention path, which only showed up on Blackwell GPUs (Hoppers were clean) and even there only a fraction of the time.

You can’t read a bug like that out of the source, and you can’t catch it in a single run. So we turned on heavy debug logging aimed squarely at those spikes and, the instant a step’s grad norm blew up, snapshotted the exact input and weights that caused it, then replayed that frozen input through the same update dozens of times and counted how often it came out wrong. That gave us a rate to measure against and one real clue: only the larger, densely packed batches ever triggered it. The cause was a race condition: the kernel chose how much parallelism to use at runtime, one of those choices was unsafe on Blackwell GPUs, and so the identical input could come out clean on one run, and corrupt on the next. The fix was one line (pinning the kernel to a safe setting), but finding it took days of replaying the same input and counting.

A quieter one lived in sequence packing. Packing many short rollouts into one row is standard for throughput, and the attention layers handle it the usual way, with a varlen mask (cu_seqlens) that stops one sequence from attending into the next. The linear-attention path is where it gets subtle: a recurrent-state layer has no attention mask to lean on, so it has to explicitly reset its state and causal convolution at each sequence boundary. We found a bug causing the state to bleed across packed sequences. There was no crash, just quietly wrong gradients. The model trained, only a little bit wrong, and nothing told us.

The lesson from both is to watch the numerics as closely as we watch the reward, and to never explain away an inaccuracy just because it didn’t take the run down with it. So we ask numerical questions constantly. Does the probability the trainer computes for a token match the one the rollout actually sampled? Do the gradients agree with sequence packing on and off, across precisions, across different parallelism layouts? We wire those comparisons into the run itself, and when one drifts, even slightly, even only sometimes, we treat it as a bug to find rather than a number to smooth over.

3. Have a weight-sync strategy

We run fully async RL: generation and training run at the same time, so the GPUs never sit idle waiting for each other. But at the instant we swap, thousands of requests are somewhere in the middle of generating, and what we do with those in-flight requests turns out to be a real decision, with real costs on every side. The space of solutions is well documented, but neither the choice nor the implementation is obvious: the options trade off in subtle ways, and each takes real care to implement correctly. vLLM’s native RL APIs, HuggingFace’s survey of open-source RL libraries, and the PipelineRL and AReaL papers are a good place to read up.

It helps to see this as two separate decisions that tend to get blurred together. The first is about the work a request has already done, i.e. the tokens it has generated so far. The second is about that work’s KV cache, i.e. the intermediate state the engine keeps around so it doesn’t recompute the whole sequence on every step. Keeping the two apart makes the design space fall out cleanly.

The first fork: what happens to the work in progress? There are three high-level strategies.

Continue. Snapshot the engine exactly as it is, swap the weights underneath it, and let every request pick up generating right where it left off, now under the new weights. Nothing is thrown away and nothing waits. The price is that a single completion becomes a blend of two policies: its earlier tokens came from the old weights, its later tokens from the new ones.

Let complete. Leave the in-flight requests alone, let them finish on the old weights and only then swap. There are alternative stopping points to choose from. Two that make sense are finishing only the in-flight step or finishing the entire multi-step rollout. Choosing a finer boundary (e.g. the in-flight step) means less waiting and smaller risk of the odd extremely long completion stalling the flow.

Rewind. Cut the in-flight request, throw away the work in progress, and redo it under the new weights. Again you could make different choices how far back to rewind, where discarding only the last completed turn reduces wasted compute.

The second fork: what happens to the KV cache? This one comes in any strategy that carries already-generated tokens across the swap. Those tokens have a KV cache that was computed under the old weights, and you get to choose.

Reuse it. Keep decoding straight over the cache the old weights built. It is free, but the new weights are now attending over keys and values a different policy wrote, which piles extra off-policyness on top of what async already carries.

Rebuild it. Recompute the cache for the retained tokens under the new weights, a fresh prefill. Clean and more on-policy, at the cost of that prefill.

Each of these forks opens onto more of them. Within continue, do you pause the engine just long enough to load the weights, or swap the weights in the gap between one decoding step and the next, so generation never actually stops (vLLM’s pause-and-resume and PipelineRL’s in-flight updates, respectively)? Within let complete, do you let rollouts continue to max length if needed or do you cap the wait with a timeout and drop whatever overruns? Within rewind, how much requeue bookkeeping are you willing to carry? Every answer is another small decision with its own price tag.

For us, the shape that fell out was to let requests complete, but only to the turn boundary, cap that wait with a timeout, drop the overruns, and rebuild the KV fresh for the next turn. Within any single turn, then, every token comes from one set of weights, but a longer rollout can still mix a few versions together. Other choices are entirely possible, and create different trade-offs in terms of GPU utilization, off-policy drift, implementation simplicity and observability. The only real mistake is picking a strategy without thinking through the trade-offs.

4. The gateway should be a pane of glass

On paper, the model gateway is the simplest part of the RL stack. Ours began as a fork of rLLM‘s implementation and evolved to deal with the challenges we describe here. The gateway is a thin web server that forwards model calls between the agent rollouts and the fleet of inference GPUs that serve them, while storing the trajectory tokens and probabilities to train on later. The only other requirement is “sticky routing”: since the KV cache is GPU-local, consecutive requests from a given rollout should be routed to the same GPU to maximize cache hits. Any side effects or logic that this entails should be invisible to the agent rollouts. The gateway should be a pane of glass; you look straight through it and never think about it.

In practice, keeping the gateway transparent becomes its own small engineering challenge as we scale up the training. Consider production scale, with thousands of concurrent rollouts of long-horizon tasks like pentesting, where each request has a context length of hundreds of thousands of tokens. There are two main risks in this scenario:

1. Blocking CPU operations per request that scale with the request size. These seem negligible when developing at small scale, but as the RL runs scale up, the waiting intervals add up and grow longer, slowing down or stalling the gateway and consequently the entire training process.
   A classic example is parsing the request body into a JSON object and serializing it back out to add or remove a couple of keys for routing or logging. That body is the entire conversation so far, which grows every turn, so a two-key edit means deserializing and re-serializing hundreds of MB per second. Not science fiction, just something that would stall the event loop on a small, single-threaded Python server, which was the original implementation we inherited.
2. I/O bottlenecks on the trajectory database. The gateway stores every trajectory in a local SQLite database, which we initially ignored until it grew to ~100GB, of which ~98% were dead free pages. Every commit was checkpointing a write-ahead log across that bloated file, and that I/O landed, of course, on the event loop. Again, enough to suffocate the naive implementation we had in place.

Much like numerical errors, when these risks materialize, nothing crashes. All the dashboards stay green, but the throughput grinds to a halt. To figure out what was going on, we added a small probe to watch how far behind the event loop is falling, and when it overshoots it fires py-spy from a side thread to grab a native stack of every thread. Once we realized what was happening, we thoroughly stepped through the hot path and cleaned up any blocking CPU operations and any potential I/O bottlenecks.

The fixes were mostly small, but the impact was dramatic. To address the JSON round-trip mentioned above, we stopped parsing giant blobs we never needed to read and instead spliced the few keys straight into the raw bytes, leaving the payload untouched. An O(body size) operation became O(a few keys), and the stalls vanished. To resolve the I/O bottleneck, we stopped storing redundant fields in the trajectory database and added an occasional reclaim operation to return freed space to the OS and keep the file size manageable.

After that it was a run of smaller versions of the same lesson. Each was tiny on its own, and each stalled training once we hit sufficient scale. The rule we settled on is simple, and we hold to it strictly: the gateway does no real work on the hot path. Route the request, stream the response back, and record everything after the fact. The model should never wait on it.

5. Environments can cost more than model inference and training

Here’s the one nobody warned us about. The environments, the live web apps our agent trains against, ended up costing more than the GPUs.

Each rollout needs its own unique instance of the target app to work with: a fresh, isolated environment. We spin one up, let the agent work on it, and tear it down when the episode ends. For efficiency, we acquire and free the corresponding compute resources dynamically, but we also intentionally pre-warm target apps ahead of their rollouts, to make sure that GPUs don’t sit idle waiting for the environment to set up. 

All in all, there are thousands of apps running concurrently to support a single RL run, and the cost of that fleet adds up. In one campaign, the target apps came to somewhere between 50 and 70 percent of the total bill. The remainder was of course the GPUs, which are the expected cost driver, but they only came in second place.

So budget for your environments like you budget for compute. If you only count the GPUs, you’re likely undercounting by a significant amount.

For more about how we spin up these training environments, check out this blog post.

Conclusion

An RL pipeline is a complex system with many moving parts. And all of them are moving fast: the models, the harnesses, the frameworks, and the long tail of bugs in libraries we don’t even own. Hard won victories of the past can erode quickly as these components evolve, so the only way forward is to focus on the deeper lessons and always continue adapting. 

Over the past few months we’ve learned a lot working through the problems described here, and others that we didn’t have space to cover. We got better at knowing where to look, distrusting the system in the face of a green dashboard, and keeping the machinery out of the model’s way. We’re sharing the lessons and habits we’ve picked up in the hopes that others will find them interesting and helpful.