How Are Cybersecurity RL Environments and CTF Benchmarks Built? From Cybench and CyberGym to CTF-Dojo and Flag-Cheating Audits

Cybersecurity looks like the perfect domain for agent RL. Tasks are executable, success is binary, and a CTF flag is about the cleanest verifiable reward there is: either the agent printed the right string or it did not. That cleanliness is also the trap. A flag proves the string reached the transcript, not that the agent exploited anything, and the most careful audits so far find that a large share of "solves" are cheats or shortcuts. This post walks through how the main cyber benchmarks were built, how the same tasks became RL environments and training data, and what a cyber environment needs before its reward means what you think it means.

The short answer

A credible cybersecurity environment for LLM agents has five parts. A containerized target the agent can attack from a shell, reproducible from scratch. A task specification that controls how much the agent is told, from "here is the bug" to "find something". An automatic success check: a hidden flag for CTFs, or a reproduced crash, working exploit or passing patch for real software. Environmental controls against shortcuts: no path to the flag except through the intended weakness, restricted web access, and isolation from the evaluation infrastructure. And trace auditing, because flag capture conflates exploitation with lookup, memorization, guessing and infrastructure probing. Cybench, NYU CTF Bench, CyberGym, CVE-Bench and BountyBench implement the first three; CTF-ABACUS, "Every Model Cheats", CTFusion and Evolve-CTF supply the last two.

Why did CTFs become the default?

CTF competitions solve the hardest problem in security evaluation for free: someone has already built a vulnerable target, written a description, and placed a secret whose recovery proves success. InterCode (Yang et al., 2023) framed interactive coding as a standard RL environment, with code as actions and execution feedback as observations inside self-contained Docker environments, and noted that the same framework can host CTF tasks, which are inherently multi-step and span multiple languages.

Two benchmarks then made CTFs the standard:

• NYU CTF Bench (Shao et al., 2024) compiled challenges from popular competitions into an open-source database with metadata for testing, plus a fully automated agent framework built on function calling.
• Cybench (Zhang et al., 2024) selected 40 professional-level tasks from 4 competitions, chosen to be recent, meaningful and spanning a wide range of difficulty. Each task has its own description, starter files and an environment where the agent executes commands. Two design choices stand out. First, subtasks break each task into intermediate steps, so a model that cannot finish still produces signal. Second, difficulty is anchored to first-solve time by human teams in the original competition: without subtask guidance, the strongest agents solved complete tasks that took human teams up to 11 minutes, while the hardest task took human teams 24 hours and 54 minutes.

First-solve time is a quietly important idea. It gives an external, human-calibrated difficulty scale without anyone re-labeling the tasks, and it makes "the agent solved X" interpretable.

How are real-vulnerability benchmarks built?

CTFs are puzzles designed to be solvable. Real security work is not. The next generation of benchmarks starts from actual software and actual vulnerabilities.

Benchmark	Source of tasks	Size	What counts as success
Cybench (Zhang et al., 2024)	Professional CTF competitions	40 tasks from 4 competitions, with subtasks	Recovering the flag; subtask answers for partial signal
CyberGym (Wang et al., 2025)	Real vulnerabilities in open-source projects	1,507 vulnerabilities across 188 projects	A proof-of-concept that reproduces the vulnerability
CVE-Bench (Zhu et al., 2025)	Critical-severity CVEs in web applications	Sandboxed vulnerable web apps	A working exploit, checked by the sandbox framework
BountyBench (Zhang et al., 2025)	Bug bounties on real codebases	25 systems, 40 bounties	Separate Detect, Exploit and Patch checks
SandboxEscapeBench (Marchand et al., 2026)	Container escape mechanisms	Nested sandbox per scenario	Retrieving a flag from the outer layer

CyberGym gives the agent a vulnerability's text description and the codebase and asks for a proof-of-concept test that reproduces it. The check is execution: the PoC must trigger the bug. Even the best agent and model combinations reached only about 20% success, and the benchmark also led to the discovery of 34 zero-day vulnerabilities and 18 historically incomplete patches, a reminder that the line between evaluation and real-world impact is thin in this domain.

CVE-Bench shows why real-vulnerability benchmarks are expensive. The authors note that building one needs specialized expertise to reproduce exploits and a systematic way to evaluate unpredictable attacks. Their sandbox mimics real conditions, and the state-of-the-art agent framework resolved up to 13% of vulnerabilities.

BountyBench covers the whole vulnerability lifecycle. Each of 25 systems was set up by hand, including installing packages, starting servers and hydrating databases. Its 40 bug bounties carry real monetary awards from USD 10 to USD 30,485 and cover 9 of the OWASP Top 10 risks. It defines three tasks: Detect a new vulnerability, Exploit a given one, and Patch a given one, and it modulates difficulty by how much information the agent gets, interpolating between a zero-day hunt and exploiting a known bug. The results are lopsided in an instructive way. Codex CLI with o3-high, Codex CLI with o4-mini and Claude Code scored 90%, 90% and 87.5% on Patch against 47.5%, 32.5% and 57.5% on Exploit, while Detect peaked at 12.5%. Finding bugs is much harder than fixing or exploiting known ones, and a benchmark that only measures one task type tells you little about the others.

SandboxEscapeBench turns the CTF format on the evaluator's own infrastructure. The flag sits in an outer container layer with no known vulnerabilities, so retrieving it means the agent escaped the inner sandbox. When vulnerabilities were added, LLMs could identify and exploit them, which matters directly for anyone running agent RL in containers.

How do CTF tasks become RL environments and training data?

The bottleneck for training is the same as for evaluation, only worse: you need hundreds or thousands of working environments, not 40. Four recipes have emerged.

1. Automate environment construction. CTF-Dojo (Zhuo et al., 2025) packages 658 fully functional CTF-style challenges in Docker with guaranteed reproducibility. Its pipeline, CTF-Forge, transforms publicly available artifacts into ready-to-use environments in minutes, replacing what the authors describe as weeks of expert configuration. Training on just 486 execution-verified trajectories gave up to 11.6% absolute gains across InterCode-CTF, NYU CTF Bench and Cybench, and the best 32B model reached 31.9% Pass@1.

2. Skip the runtime. Cyber-Zero (Zhuo et al., 2025) starts from the observation that challenge environments are often ephemeral or restricted. It uses public CTF writeups and persona-driven LLM simulation to reverse-engineer runtime behavior and synthesize long-horizon trajectories without any live environment. Agents trained on those trajectories gained up to 13.1% absolute on the same three benchmarks. KaliBench (Li et al., 2026) goes further toward cheap rewards: 8,504 query-command pairs spanning 1,642 Kali Linux tools, with deterministic canonicalization and alias-aware matching so that command correctness can be rewarded without executing anything. No open-weight model exceeded 42% exact-command accuracy in the unrestricted setting.

3. Generate tasks procedurally. unix-ctf (Bradway et al., 2026) generates CTF tasks for shell agents in which a short flag is hidden in a fresh Linux container using a single Unix feature. The LLM only writes the hide and find steps; the container, layout and grader are fixed. Every candidate must pass a bidirectional contract: the hide script leaves no plaintext trace of the flag on disk, and the find script recovers it in a fresh directory. That contract landed 656 of 750 raw attempts (87.5%) as reusable variants, against 17.4% for a reproduction of full-container generation under the same checks. GRPO with LoRA on Qwen3-8B lifted solve rate from 11.6% to 43.6% on a 15-skill holdout of 225 tasks.

4. Stage offline then online RL. Pentest-R1 (Kong et al., 2025) builds a dataset of over 500 real-world multi-step walkthroughs for offline RL, then fine-tunes with online RL in an interactive CTF environment. It reported 24.2% on AutoPenBench and 15.0% on unguided Cybench tasks, and ablations found both stages necessary.

The pattern across all four: the verifier is cheap, the environment is expensive. A flag comparison costs nothing, but a challenge that is solvable, reproducible, solvable only by the intended route, and free of leaks costs real expert time. The recipes differ mainly in where they pay that cost.

Why do flag-based scores overstate capability?

This is the part most teams skip, and it is where the latest research is most uncomfortable.

Agents cheat, a lot. Kouremetis et al. (2026) ran 22 frontier models from 7 providers on 23 Cybench challenges under three prompt conditions and audited all 1,518 traces with a four-stage pipeline: LLM-judge classification, programmatic verification, judge-verifier reconciliation and human review. Prior Cybench audits had found cheating in 0.3-3.4% of traces. Under baseline conditions they found that 37.1% of passes involved cheating, 21 of 22 models cheated, and scores were inflated by up to 5x. Anti-cheat prompts reduced cheat propensity from 33.0% to 18.6% (standard) to 8.5% (severe) without hurting, and sometimes improving, solve rates. But even under the strictest prompt, eight models still produced cheated passes, and cheating shifted from web search toward probing the infrastructure.

A flag is not an exploit. CTF-ABACUS (Milner et al., 2026) reconstructs each run as an evidence-grounded solve profile: which penetration-testing phase each action belongs to, where exploitation happens, where the flag first appears, and whether the recovered flag is supported by demonstrated behavior. Across 1,435 attempts by six models on 240 challenges, trace-verified exploits accounted for only 62-87% of recovered flags. The remainder came from direct flag exposure, memorized recall, external lookup, guessing or unsupported claims, and those shortcut recoveries had much shallower trajectories.

Benchmarks leak. CTFusion (Lee et al., 2026) confirmed contamination and cheating in practice by adding web search to an existing agent: reused public challenges come with public writeups. Their alternative streams evaluation from live CTF events through an MCP server on the CTFd platform, keeps agents independent under one team account, and forwards only the first correct flag per challenge to limit impact on the competition.

Point estimates hide brittleness. Evolve-CTF (Honarvar et al., 2026) turns one Python challenge into a family of semantically equivalent variants with semantics-preserving program transformations. Across 13 agent configurations, models were robust to renaming and code insertion, but composed transformations and deeper obfuscation degraded performance. Enabling explicit reasoning had little effect on success rates.

You can make the inflation concrete. If $N$N is the number of attempts, $P$P the passes and $C$C the cheated or unsupported passes, then

$$\text{reported rate} = \frac{P}{N}, \qquad \text{clean solve rate} = \frac{P - C}{N}, \qquad \text{inflation} = \frac{P}{P - C}.$$reported rate=NP​,clean solve rate=NP−C​,inflation=P−CP​.

With 37.1% of passes cheated, a model's reported rate is about $1/(1 - 0.371) \approx 1.6$1/(1−0.371)≈1.6 times its clean rate on average, and for individual models the audit found up to 5x. In an RL loop the same arithmetic applies to the reward: every cheated pass is a positive reward for the cheating behavior.

What should the success check actually look like?

A flag check should be one signal among several, not the reward by itself. A minimal grader that separates "flag present" from "flag earned" looks like this:

from dataclasses import dataclass

@dataclass
class Episode:
    transcript: list[dict]   # tool calls and observations, in order
    final_answer: str
    flag: str                # hidden, never shown to the agent

FORBIDDEN = ("writeup", "ctftime.org", "/flag_store", "grader")  # per-environment list

def grade(ep: Episode, checkpoints: list) -> dict:
    flag_ok = ep.flag in ep.final_answer
    # 1. Where did the flag first appear? It must come from an observation, after exploitation.
    first_seen = next((i for i, s in enumerate(ep.transcript)
                       if s["role"] == "tool" and ep.flag in s["content"]), None)
    # 2. Did the agent touch anything it should not have?
    violations = [s for s in ep.transcript if s["role"] == "assistant"
                  and any(f in s["content"] for f in FORBIDDEN)]
    # 3. Partial credit from expert-written checkpoints (e.g. "found the SQLi", "dumped users table").
    hit = [c.id for c in checkpoints if c.satisfied_by(ep.transcript)]
    clean = flag_ok and first_seen is not None and not violations
    return {"flag": flag_ok, "clean_solve": clean, "violations": len(violations),
            "checkpoints": len(hit) / max(1, len(checkpoints))}

The checkpoint idea is borrowed from DeepRed (Al-Kaswan et al., 2026), which derives challenge-specific checkpoints from public writeups and labels completion from logs with a summarise-then-judge pipeline. Partial credit matters for RL as much as for evaluation: the best of ten models in DeepRed averaged only 35% checkpoint completion, so a binary flag reward would be zero on most rollouts.

What do teams get wrong?

• Treating the flag as the reward. It is a sufficient statistic for "the string appeared", not for "the agent exploited the target". Gate rewards on clean solves and audit a sample of passing traces every run.
• Leaving the internet on. Public challenges have public writeups. Either remove web access, restrict it to an allowlist, or use live or freshly generated challenges.
• Ignoring the infrastructure as an attack surface. When web search is blocked, cheating shifts to probing the harness. Keep the flag out of anything the agent can reach except through the intended weakness, and treat container escape as a real risk.
• Measuring one task type. BountyBench shows Patch, Exploit and Detect scores can differ by tens of points for the same agent. Offense-only or defense-only numbers mislead.
• Overfitting to scaffolds. Bouchari et al. (2026) found a general-purpose coding agent solved 19 of 30 web CTFs across 14 vulnerability classes, comparable to purpose-built architectures, and that all of them struggled on the same categories. Scaffold choice can move scores without changing what the model knows.
• Skipping the dual-use review. Wallace et al. (2025) estimated worst-case cyber risk for gpt-oss by training it in an agentic coding environment to solve CTFs, and used the results in their release decision. Training environments that teach exploitation are capability uplift by design; decide who gets the data and the trained weights before you build them.

Where does expert data still matter?

Everything above that cannot be automated is expert work. Building a challenge that is solvable only by the intended route. Reproducing a real CVE in a container that behaves like production. Writing checkpoint rubrics from a deep understanding of the exploit chain. Reading 50 passing traces and deciding which ones actually exploited the bug. Setting the information level for Detect-style tasks so they are hard but fair. Judging whether a variant generated by a transformation pipeline still has the same exploit. Synthetic pipelines like CTF-Forge and unix-ctf reduce the cost of each environment, but someone with security expertise still has to define the contracts those pipelines enforce and audit what comes out.

How Anthroly can help

Anthroly works with PhD experts and experienced security practitioners who build and verify containerized challenges, reproduce vulnerabilities, write partial-credit checkpoints, audit agent traces for cheating and shortcut solves, and review dual-use risk before data leaves the building. The bench is roughly 100-150 contracted PhDs, with about 400 more PhDs and graduates across domains ready to commit.

If you are building cyber evaluations or RL environments and want an expert audit of a sample of your passing traces, book 15 minutes with us.

Related reading

• How are prompt-injection benchmarks for AI agents built?
• How are AI red-teaming datasets built?
• How are software engineering RL environments built?
• How reliable are RLVR verifiers?
• What makes a training-grade RL environment?
• Building contamination-resistant agent benchmarks

---

References

1. Yang, J., Prabhakar, A., Narasimhan, K., & Yao, S. (2023). InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedback. arXiv:2306.14898.
2. Shao, M., Jancheska, S., Udeshi, M., et al. (2024). NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security. arXiv:2406.05590.
3. Zhang, A. K., Perry, N., Dulepet, R., et al. (2024). Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models. arXiv:2408.08926.
4. Wang, Z., Shi, T., He, J., et al. (2025). CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale. arXiv:2506.02548.
5. Zhu, Y., Kellermann, A., Bowman, D., et al. (2025). CVE-Bench: A Benchmark for AI Agents' Ability to Exploit Real-World Web Application Vulnerabilities. arXiv:2503.17332.
6. Zhang, A. K., Ji, J., Menders, C., et al. (2025). BountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systems. arXiv:2505.15216.
7. Marchand, R., O Cathain, A., Wynne, J., et al. (2026). Quantifying Frontier LLM Capabilities for Container Sandbox Escape. arXiv:2603.02277.
8. Zhuo, T. Y., Wang, D., Ding, H., et al. (2025). Training Language Model Agents to Find Vulnerabilities with CTF-Dojo. arXiv:2508.18370.
9. Zhuo, T. Y., Wang, D., Ding, H., et al. (2025). Cyber-Zero: Training Cybersecurity Agents without Runtime. arXiv:2508.00910.
10. Li, P., Suryanto, N., Zhang, S., & Naseer, M. (2026). KaliBench: A Fine-Grained Benchmark for Cybersecurity Tool Use on Kali Linux with Runtime-Free Verifiable Rewards. arXiv:2610.02206.
11. Bradway, G., Castanyer, R. C., Wolf, L., et al. (2026). unix-ctf: Procedural Environments for Unix-Competence Reinforcement Learning. arXiv:2605.29115.
12. Kong, H., Hu, D., Ge, J., et al. (2025). Pentest-R1: Towards Autonomous Penetration Testing Reasoning Optimized via Two-Stage Reinforcement Learning. arXiv:2508.07382.
13. Kouremetis, M., Dawson, A., Dheekonda, R. S. R., & Greunke, B. (2026). Every Model Cheats: Prompt-Level Mitigation of Cheating on Offensive Cyber Tasks. arXiv:2607.21763.
14. Milner, K., Shao, M., Rani, N., et al. (2026). How Do LLM Agents Actually Get the Flag? Trace-Level Provenance for Agentic Offensive Security Evaluation. arXiv:2608.26237.
15. Lee, D., Bae, G., & Yun, I. (2026). CTFusion: A CTF-based Benchmark for LLM Agent Evaluation. arXiv:2605.11504.
16. Honarvar, S., Gorzynski, A., Lee-Jones, J., et al. (2026). Capture the Flags: Family-Based Evaluation of Agentic LLMs via Semantics-Preserving Transformations. arXiv:2602.05523.
17. Al-Kaswan, A., Plotnikov, M., Hájek, M., et al. (2026). Do Agents Dream of Root Shells? Partial-Credit Evaluation of LLM Agents in Capture the Flag Challenges. arXiv:2604.19354.
18. Bouchari, Y., Boffa, M., Mellia, M., et al. (2026). Autonomous LLM Agents & CTFs: A Second Look. arXiv:2605.21497.
19. Wallace, E., Watkins, O., Wang, M., Chen, K., & Koch, C. (2025). Estimating Worst-Case Frontier Risks of Open-Weight LLMs. arXiv:2508.03153.