Small Language Models (SLMs) for offensive security

Closing the gap to frontier model accuracy at a fraction of the cost by fine-tuning small, open-weights language models specifically for offensive security.

Why small models?

State-of-the-art LLMs are increasingly capable drivers for offensive security agents. Both closed frontier models and the recent, largest open-weights models (e.g. GLM5.2, Kimi K3) continue to push the cybersecurity capability ceiling. Yet these models remain slow and expensive to run at scale. Using frontier LLMs, the cost of providing continuous testing coverage for large engineering teams can quickly skyrocket, motivating the search for more efficient and cost-effective alternatives.

In this post we explore one such promising direction: leveraging smaller models^[1]. We take this proposition to the extreme, focusing on open-weights models up to roughly 30B parameters. For a sense of scale, the models we examine here are 50×-400× smaller than frontier LLMs. These models are simple and affordable to train and serve at scale, even in resource-constrained on-prem deployments lacking massive GPU clusters.

We evaluate five models from two families, Google DeepMind’s Gemma-4 ^[2] and Alibaba Cloud’s Qwen-3.5 ^[3], on verifiable web vulnerability exploitation tasks, comparing the labs’ stock weights against our own fine-tuned versions of the three Qwen-3.5 variants. Alongside fine-tuning, we apply test-time compute scaling and report results in terms of cost-accuracy tradeoff.

Three questions frame the rest of this post: how accurate are small open models out-of-the-box (without additional training), does targeted fine-tuning close the gap to the frontier, and is the result genuinely cheaper in practice?

The task: XSS exploitation

Offensive security workflows can be divided into various stages and subtasks. For the purposes of this report we choose to focus on XSS vulnerability exploitation, where the agent is required to validate a hypothesis claiming a specific XSS vulnerability in a web application. The agent is given access to a running instance of the web application and the hypothesis, which is phrased in natural language and can vary in accuracy and specificity. The agent is required to validate the hypothesis by producing a working exploit based on its interaction with the live app. The task is completed once the agent submits an exploit proof-of-concept (POC) script or declares a “not exploitable” verdict. A deterministic validation engine then replays the POC script in a clean environment and rules ‘pass’ or ‘fail’, based on whether the exploit fired against the real app.

The XSS exploitation task, in its specific formulation above, is a valuable laboratory for a number of reasons. First, XSS is a frequent finding in security testing. Findings without a proof of exploit are indistinguishable from noise, so fast and accurate validation is highly impactful for practitioners. Second, XSS can be deterministically validated, so we can implement an LLM-free eval protocol for this task and significantly reduce evaluation noise. Third, once we build a suitable deterministic validation engine, we can incorporate it as part of our test-time scaling strategy.

Evaluating small open models out-of-the-box

We start by evaluating 5 open-weights models: Gemma-4 31B, Qwen-3.5 27B, Gemma-4 12B, Qwen 3.5 9B and Qwen 3.5 4B. For each model we used the stock instruction-following weights and the provider-recommended sampling parameters (temperature, top-p, top-k, etc.). The same agent harness is used across all the models.

We collected an evaluation set with 100 unique (web application, hypothesis) pairs, where all the hypotheses are verified true positives. The infrastructure for creating and serving these apps is described in detail in our previous blog post. We evaluate each model’s exploit rate — the percentage of true vulnerabilities it was able to correctly validate — and compare the open models to the closed frontier, represented by Claude Opus 4.8, which solves 96 of the 100 targets on its first attempt.

Our harness for this task supports two test-time scaling levers:

1. Depth-wise scaling (context length): The agent is provided a submit_poc tool that it can call multiple times, so the agent can inspect a failed POC, apply reasoning, collect more information from the live app, improve the POC and resubmit. The agent can continue trying until it either submits a working POC script or runs out of context budget. The maximum allowed context length is the natural knob for depth-wise compute budget scaling.
2. Breadth-wise scaling (pass at k): We run k>1 independent instances of the agent. As soon as one of the agents submits a working POC script, we terminate. The maximum number of independent attempts, k, provides a natural knob for scaling the compute budget breadth-wise.

The results of our evaluation are shown in Figure 1. We report pass@k, the chance that k attempts include at least one successful exploit, using an unbiased estimator ^[4]. Depth-wise scaling is held constant throughout these evaluations.

The strongest open-weight models we tested — Gemma-4 31B and Qwen-3.5 27B — land near 55% pass@3, and Qwen-3.5 27B tops out at 63% when given up to 9 attempts per example. The field drops off quickly from there: the 9B and 12B models are vastly inferior and the 4B model has practically zero capability. The takeaway is that the best open-weight bases are promising, most lag well behind, and none are frontier-level without additional improvements.

Fine tuning open models

The pre-trained model evals show promise but still lag behind the frontier. To close the gap, we fine-tune each model using our proprietary training recipe (read more about our in-house training infrastructure here). For SFT data, we used distillation with rejection sampling using a stronger teacher model, meaning we ran multiple exploitation attempts with the teacher and kept only confirmed successful traces. The final training set included approximately 2,000 examples of trajectories producing a successful exploit. We used the same training set for all evaluated models and checked it carefully to rule out any accidental leakage or overlap with the evaluation set.

We used parameter-efficient fine-tuning, specifically LoRA (Low-Rank Adaptation) ^[5]. The same fine-tuning recipe is applied to all models, with adjustments to hyperparameters such as learning rate and LoRA rank to optimize individual model performance fairly.

Fine-tuning significantly improves performance for all models. The impact varies by the size of the student and the headroom left on top of the accuracy of the stock post-trained weights. For the two smallest models, our recipe lifts capability dramatically starting from near-zero bases: the Novee fine-tuned 4B parameter model improved from 2% to 51% pass@3, and the 9B model improved from 10% to 56% pass@3. Models that were nearly useless out of the box become genuinely capable exploit agents. The gains extend to larger and more capable models as well. Qwen-3.5 27B starts at 55% pass@3 for the stock weights and reaches 82% with finetuning.

The 27B fine-tune also converts the hardest part of the eval set from failures to successful exploits. Given a larger retry budget it reaches 85% pass@5 and 87% pass@9, against 63% for the stock weights at their full budget, effectively reducing false negatives by 3×. What remains are the longest-horizon vulnerabilities in the evaluation set: 13 of the 100 targets are never solved in nine attempts.

Fine-tuning also improves efficiency, independently from exploit success rates. Figure 3 shows the trajectory length statistics for successful attempts by Qwen-3.5 27B. The fine-tuned model needs about 20% less peak context per solve (median 31k versus 39k tokens for the base) while taking essentially the same number of steps (median 32 versus 30). This is not the model doing less work: the reasoning within each step becomes more succinct and focused, and the context it saves is what shows up as a lower cost per solved target in Figure 4.

Another phenomenon we observe is that fine-tuning improves persistence. The base models give up more frequently after exploring a target. The fine tuned models push through, submit more attempts and persist to solve hard targets that the base never cracks. This includes four genuine zero-days in real open source applications.

Pareto frontier using fine-tuned models

In practice, there is a tradeoff between performance and cost. In Figure 4 below we show how fine-tuning for pentesting tasks helps beat the tradeoff offered by general-purpose models. Using only general purpose models, one is left with choice between the most expensive frontier models with high exploit rates and cheap small open models with low exploit rates. Our fine-tuned models have the same low cost of the base open weights models, but deliver vastly superior exploit rates. For example, using a fine-tuned 27B parameter model we achieve ~80% of the exploit rate of the frontier at ~40% of the cost.

The cost figures for the open and fine-tuned models are deliberately conservative: We did not optimize inference at all, and simply used standard API provider pay-per-token prices. A dedicated in-house deployment can be optimized to achieve much more substantial cost reduction at scale. We leave this outside the scope of this blog.

Conclusion

We sum up our experiments with overall positive conclusions. Although current small open-weight models are not yet strong enough to compete with frontier models out-of-the-box, they are sufficiently capable to serve as bases for fine-tuning. With targeted training, we are able to close most of the gap between open and closed models, at a dramatically lower inference cost. Those savings are measured on standard API prices; dedicated in-house serving can push them further. The resulting models can be deployed on-prem in contexts that require strict data privacy, even without allocating massive compute clusters. For a focused offensive-security task, fine-tuning a small open-weight model is a genuine cost and performance alternative to a frontier API, and it makes broader, more frequent agentic penetration testing coverage possible.

References and Footnotes

1. Often called SLMs (Small Language Models). Opinions differ on the parameter count threshold for a language model to qualify as “small” or “large”. ↩
2. Gemma Team, Google DeepMind. Gemma 4 Technical Report. 2026. arXiv:2607.02770 ↩
3. Qwen Team. Qwen3 Technical Report. 2025. arXiv:2505.09388 ↩
4. Chen, M., Tworek, J., Jun, H., et al. Evaluating Large Language Models Trained on Code. 2021. Introduces the pass@k unbiased estimator. arXiv:2107.03374 ↩
5. Hu, E., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. LoRA: Low-Rank Adaptation of Large Language Models. 2021. arXiv:2106.09685 ↩