Today A Security is introducing ARENA, the Agentic Red-team Evaluation for Neutral Assessment. The ARENA platform helps red teams, pentesters, security researchers, and enthusiasts generate fresh, real-world-shaped vulnerable targets with ground truth, so they can measure an AI attacker’s performance, sophistication and results over time.
The buyer's problem in 2026
Weaponized AI has changed the economics of attacking enterprises. Frontier models got good at offensive security faster than anyone expected. Today, a single sophisticated attacker, enhanced by top models, can quickly achieve what previously required months of research and a dedicated team. Automated red teaming that keeps pace with the rising risk is becoming the norm. Indeed, the easy part of hacking can now be automated. It means matching that ability isn't enough; it's the new baseline.
Every product looks capable against a target that doesn't resist. Comprehensiveness and scale are where that stops being true, and there's a good reason they're the differentiator: they're hard. An adversary needs one opening, which means a defense can't leave anything to luck - full coverage of a sprawling surface, hundreds of routes, layered auth, defenses in the way, with findings surfaced reliably and in volume.
School vs Life
When the goal is to leave no stone unturned - it raises another problem: how should these solutions be evaluated?
Point them at production, and you own the fallout: agents running wild and causing harm, plus a coordination tax with every team that owns a service. Staging environments rarely resemble prod, and neither have a ground truth.
Some teams reach for CTFs, off-the-shelf or wired up in-house with hand-built scoring. The problem with CTFs and known challenges is that they were initially built to teach. They isolate specific cases, and miss the real-world chaos. They're high-school tests. Acing one says little about the job.
The 94% morning, and the 11% afternoon
A couple of months into building our agent, we ran it against WebGoat, a popular open-source CTF, and it scored 94%. We celebrated for about an hour.
That afternoon we pointed it at a snapshot of a real-world e-commerce application. Nothing exotic: just a few hundred routes, a real auth flow, a WAF in front, the usual smell of production. It scored 11%, measured against a manual test we ran ourselves, since a live app has no answer key of its own. Against the bugs our own researchers had found manually assisted by AI, our agent caught barely one in ten, and buried them under a flood of false positives.
The 94% wasn't a lie. It was a memory.
Public CTFs like Juice Shop and WebGoat are so thoroughly documented (writeup, repos, walkthroughs) that the model underneath has effectively memorized the solutions. A high score there measures regurgitation, not reasoning.
What the real world actually looks like
The application at the top of your incident queue isn’t Juice Shop. Hundreds of routes spread across years of development. Separate authentication flows for customers, employees, partners, and APIs. A GraphQL endpoint the mobile team forgot to secure. WAFs, CAPTCHAs, rate limits, and MFA. And, increasingly, an LLM plugged into customer support that nobody has ever red-teamed.
That is the surface a red team agent has to operate on. So that is the surface we had to generate, on demand, at scale, with ground truth.
Tesla does not train Full Self-Driving by only running laps in Fremont. It trains in synthesized worlds - generated intersections, weather, edge-case pedestrians - because the real world is unbounded and a policy has to generalize, not memorize. Our 94% morning had the same problem shape an early FSD model has on a known route: great memory, mediocre capability. So we stopped grading on vulnerabilities alone and started grading on what the agent could actually do.
The same idea, applied to offense: a generated world the agent has never seen, with every hurdle placed deliberately, every edge case earned. Dynamic, yet consistent.
Bug bounty - almost the right answer
The strongest counterargument is to skip simulation entirely: turn the agent loose on real bug bounty programs. Real apps, real surface, real adversarial conditions. We respect it - in some ways it is more honest than any simulator.
But there is exactly one thing bug bounty cannot have: a source of truth. It tells you what was found; it can never tell you what was missed. Hits are visible, misses are invisible. That is fine for paying out hunters. It is fatal for evaluating one.
What ARENA does instead
We realized we needed a way to create real-life, transparent, repeatable benchmarks if we want to build a solution worth talking about, so we’ve built it ourselves.
ARENA is a generator, not a fixed lab.
Give it a set of capabilities and it builds a working, containerized web application with planted vulnerabilities - wrapped in the things that make a real target hard: WAFs, CAPTCHA and rate limiting, MFA and SSO and single-session guards, enterprise-scale routing.
Every planted vulnerability is documented as ground truth and validated as actually reachable before the lab ever ships. No two labs are the same, and no model has trained on them, because they didn't exist until you asked.
And the scoring stays grounded in what practitioners already trust. We still count validated vulnerabilities found - capabilities don't replace that number, they weight it. An easy XSS you reach only after evading a WAF and clearing a single-session guard is worth more than the same bug sitting in the open. The vulnerability is still the unit; the real-world conditions around it set the multiplier.
What you get out are the two numbers that are our north-star metrics we work continuously to improve:
- Recall - the share of real vulnerabilities the tool finds -
- Precision - the share of its findings that are real, not noise that needs to be triaged and validated.
Ground truth is what lets you measure either one honestly.
Benchmarking ourselves against the platform provides us with a blind, consistent yet dynamic simulation of the real world. Agents don’t know what they're going into, but we can still score them on their performance, success and failures.
Check it out at Black Hat
ARENA is open source on GitHub under MIT. Using the platform, you could generate a target none of your vendors has seen, hand them all the same one, and read the result. It’s open and transparent, so the playing field is fair. Contribute more hurdles and capabilities that represent real-world scenarios to see who can break them.
We’ll be demoing the tool at A Security’s booth number 4148 during Black Hat; swing by to check it out!
A hosted version will be released in the next few weeks, join the waiting list at https://arena-labs.ai.
A Security trains AI red team agents in generated worlds at scale, then deploys them against the real one.
Part 1 of a three-part series. Next: Capabilities, not challenges - the 23-capability framework and how the generator actually scores an attacker. Then: Which frontier model makes the best attacker?
