ARENA

Booz Allen published its Cyber Weapon Index this month, and the finding being quoted is the kill chain. Eighteen frontier models, American and Chinese, were run as autonomous attackers against production-grade enterprise networks, with every action validated against network telemetry, host logs and intrusion-detection sensors rather than accepted on self-report. All but one penetrated the network without human assistance.

The more consequential result sits underneath that headline. A model placed fifteenth in the index, scoring thirteen, matched the top-ranked model once it was connected to an attack harness: the software that supplies a model with tools, holds its attention across a long operation, and lets it recover after a failed attempt. Sixty-seven points of apparent capability turned out to belong to the software around the model rather than to the model. Booz Allen's own reading is that the model is no longer the unit of risk, and that the system is.

We agree, and the conclusion extends further than threat assessment.

The consequence for anyone buying offensive security

Every autonomous offensive security platform on the market is a harness wrapped around one or more frontier models. A security organization evaluating those platforms is evaluating combinations, not models, and no model-level ranking can tell it which combination will perform adequately in its environment. The Cyber Weapon Index demonstrates that about its own scores.

Answering the procurement question requires an instrument that can do two things: score an assembled system against a target where the correct answer is already documented, and hold one half of that system fixed in order to observe the contribution of the other. Until today no such instrument was publicly available, and each substitute fails for its own reason. Public challenge corpora are memorised, so a high score records recall of a documented walkthrough rather than offensive reasoning. Private benchmarks disclose neither contents nor method, which makes the resulting score a claim rather than a measurement. Production has no answer key and reports only what a system found, never what it missed. Staging removes the operational risk, keeps the missing answer key, and adds an assumption that staging resembles production in the respects that make offense difficult.

Freshness alone does not resolve this. A privately held, never-published challenge application is still small, single-service, undefended and organised around one solve path. It cannot exhibit whether a system evades a web application firewall, clears a multi-factor gate, survives adaptive throttling or sustains a plan across a long cross-service chain, and a behaviour absent from the target cannot be measured however sophisticated the scoring.

ARENA

ARENA generates fresh, never-before-seen vulnerable applications on demand: multi-service targets carrying real defences — web application firewalls, CAPTCHA gates, rate limiting, multi-factor authentication, single-session guards — each shipped with a validated, documented ground truth for every planted vulnerability. Because the artifact is a generator rather than a corpus, there is nothing to memorise, and the instrument does not decay as models improve.

Four things are published under the MIT license, with no account required and nothing reserved for a commercial tier.

‍The capability framework. Twenty-three offensive capabilities across four categories — Exploit Complexity (12), Defense Layering (3), Identity and Access (3), Scale and Surface (5) — each defined at four levels of sophistication. The list records hurdles encountered over more than a year of autonomous testing against several hundred real targets, and it is the rubric we score ourselves against. Capabilities compound within a lab: an easy cross-site scripting flaw reached only after evading a firewall and clearing a single-session guard is a harder problem than a difficult one sitting in the open, and it scores accordingly.

‍The generator. The engine that turns a capability specification into a working, containerized application. Every lab passes a validation gate before it ships, confirming each planted vulnerability is genuinely reachable and exploitable and that the ground truth matches the application. A lab nobody can solve is as useless as one everybody has memorized.

‍The evaluator. Scoring of any submission against the answer key, on recall weighted by capability level, precision, scope drift and cost per validated finding. A submission may be any file containing a list of findings, which is what allows a bare model, an in-house harness and a commercial platform to be compared on identical terms.

‍A contribution path. Capabilities are free-text YAML, contributed by pull request without touching generator internals. The list is explicitly incomplete.

What the instrument shows

ARENA was built to evaluate complete systems.

We ran seven frontier models through an identical pipeline against a single generated lab, a realistic ticketing application seeded at maximum difficulty.

‍

The models found real vulnerabilities we had not planted, enough that eight seeded bugs became a validated ground truth of thirty before any model was scored. Four results are worth stating here; the full method and the per-model numbers are in the companion study.

‍No model led on every axis. Different models led on coverage, precision, scope discipline, reliability and cost. Committing an entire system to one model forfeits performance on the axes where that model is weak, which is why we route across several and re-run the comparison whenever a new model ships.

‍Cost and capability are close to uncoupled. The spread from cheapest to most expensive run was nearly sevenfold, the two most expensive models finished at the bottom of the value table, and the model leading on F1 was also the cheapest.

What this changes

Vendor claims become comparable. A security organization generates a target none of its candidates has encountered, hands the same target to each, and reads the returned submissions against a shared answer key, producing three figures procurement can defend: coverage, false positive rate, and cost per validated finding. No production risk, and no dependence on vendor-supplied evidence. It also gives the analyst community a way to test the claim that separates adversarial exposure validation from earlier assessment models, which is whether a validation tool actually validates.

Accept No Limitations

Generated applications are not real applications. An ARENA score is a proxy for field performance, and its correct use is as evidence in a decision that also draws on real engagements. Every figure we publish is a property of a model operating inside one specific harness, and a different harness could plausibly produce a different ordering. The release has not been externally audited, and we would rather say so than imply a validation that has not occurred. Independent replication of the model comparison, external review of the ground truth procedure, and capability contributions from practitioners whose environments differ from ours would each do more for this than anything we can publish ourselves.

The rubric, the generator and the evaluator are published in full so that none of this has to be taken on trust. Generate a lab, run whatever you are evaluating against it, and recompute the result yourself.

Try Arena today, it’s completely free - http://arena-labs.ai/login