AI security benchmark 2026: What happens when you drop 15 AI coding agents into a real legacy codebase 

Long before “autonomous exploitation” discourse took over the industry, I was already watching ordinary coding agents perform surprisingly deep repo-scale security reviews on real projects. Then came Mythos, and the conversations around AI security agents have become polarized very quickly. 

One side treats frontier security agents as revolutionary offensive capability. The other argues that most of this is already possible with smaller or cheaper models. I think both sides miss the more practical question security teams should actually care about:

What happens when modern coding agents are dropped into legacy codebase repositories without isolated snippets, pre-selected vulnerable functions or toy repos designed for evals.

• Can they navigate large codebases without losing context?
• Can they reconstruct a real vulnerability chain across multiple files and services?
• Can they separate meaningful signals from operational noise? And most importantly, can they reason like security engineers instead of behaving like vulnerability autocomplete systems?

To find out, I benchmarked 15 models across 21 GitHub Copilot CLI agent runs against a vulnerable commit hidden inside a repository containing more than 2,000 files and roughly 350,000 lines of code.

The AI security agent benchmark: 15 AI agents, one real vulnerability, 350,000 lines of production code

The benchmark sat in a deliberately uncomfortable middle ground. I did not hand agents isolated vulnerable snippets, but I also did not ask them to autonomously execute polished exploit chains against live infrastructure.

Agents had to traverse the repository, reconstruct the vulnerable path, connect frontend exposure to backend trust assumptions, and explain the consequence clearly enough that a human reviewer would prioritize the correct remediation.

The vulnerability: How auth-boundary drift creates a real exploit chain

The vulnerability was an auth-boundary failure created through ordinary product drift. A backend API key originally started as a narrow, low-impact internal mechanism. With time, the key accumulated more responsibility across multiple services and was shipped into the browser build, where frontend-originated requests began using it directly despite the application already supporting JWT-based user authentication elsewhere.

On the backend, service-auth decorators treated possession of that static key as proof that the caller was a trusted internal service. Once browser-accessible code exposes a credential that backend systems treat as service identity, the security conclusion is already established.

The remediation workflow includes:

• Remove the service credential from browser paths,
• Move browser-originated requests onto the user-auth boundary,
• Stop treating browser-reachable static keys as trusted service identity.

The repository itself was operationally realistic and intentionally noisy. It contained:

• Python backend services,
• frontend code,
• Docker and CI/CD configs,
• YAML infrastructure,
• gRPC routing,
• Envoy configs,
• deployment scripts,
• README and local configuration examples.

Evaluation criteria: What actually separates a useful security report from a noisy one

The benchmark was never about whether a model could merely mention a vulnerability. The harder question was whether the model could reconstruct the exploit chain, understand the consequence hierarchy, and surface the issue clearly enough that a security engineer would prioritize the right remediation.

My evaluation criteria included:

• Did the model find the vulnerability chain?
• Did it understand the root cause?
• Did it explain the blast radius correctly?
• Did it prioritize the real issue instead of burying it under generic security noise?
• Could it connect frontend, backend, and infrastructure assumptions into one coherent security conclusion?

Every run used the same generic security-review prompt through GitHub Copilot CLI against worktrees pinned to the vulnerable commit.

Models tested:

• Anthropic Claude Opus 4.7
• OpenAI GPT-5.5
• OpenAI GPT-5.3-Codex
• OpenAI GPT-5.4
• OpenAI GPT-5.4 mini
• OpenAI GPT-5.2
• Anthropic Claude Sonnet 4.5
• OpenAI GPT-5 mini
• OpenAI GPT-5.2-Codex
• Anthropic Claude Opus 4.6
• Anthropic Claude Haiku 4.5
• Anthropic Claude Sonnet 4.6
• Anthropic Claude Opus 4.5
• Anthropic Claude Sonnet 4
• OpenAI GPT-4.1

My prompt included:

How 15 AI coding agents performed on a real-world auth vulnerability?

Most security evals score on bug detection which is a necessary bar but not a sufficient one. A report that finds the right vulnerability and buries it under twelve low-priority findings still fails the team acting on it. Someone has to triage that report. If the model got the priority wrong, the human cost of that mistake lands on whoever reads it first.

So the two columns that mattered here were narrower than a traditional score:

• Chain found: Did the model connect the full exploit chain end-to-end: the browser build exposing the credential, the frontend request path using it directly, and the backend service-auth trusting it as a valid service identity? Finding isolated fragments without reconstructing the full path counted as partial.
• Prioritized the real security issue: Did the model surface the exploit chain as the primary finding, or did it bury the issue beneath surrounding noise like .env cleanup suggestions, stricter ingress policies, internal gRPC hardening, or JWT startup validation?

The full weighted score covers chain reconstruction, root cause, evidence quality, blast radius, mitigation, severity calibration, safety hygiene, false positives, and useful secondary findings. But if a model aced every secondary category and missed or misframed the headline issue, no aggregate score would make that output safe to act on.

Our scores included:

• ✅ = yes
• ⚠️ = saw part of it or misframed it
• ❌ = missed it or got the point wrong.

Detailed scores: Evidence quality, blast radius, severity calibration

Each rubric category is shown as % of its own max. Score is the weighted total (0–100%) after penalties.

Six yes/no checks on the headline vuln. ✅ = met, ⚠️ = partial, ❌ = missing.

Stability under repeated runs: Which models hold up and which drift

Three models were re-run twice more (3 runs each) to test stability. Did the model find the primary vuln and place it as Finding #1?

Key findings of our AI security review

• Primary-issue isolation shows surprisingly weak correlation with model size or inference cost. Anthropic models like Opus 4.7 performed best, while smaller models such as GPT-5.3-Codex, GPT-5.4-mini, GPT-5.4, and GPT-5.5 ranked closely behind. Multiple earlier Opus and Sonnet variants, including Opus 4.5, Opus 4.6, Sonnet 4.6, and Sonnet 4, failed to consistently prioritize the headline vulnerability.
• Verbosity did not translate into accuracy. Opus 4.6 produced the longest report at 804 lines with 47 findings, yet lost points because of severity inflation, including 11 separate “Critical” issues, alongside an overstated XML XXE claim. By comparison, the strongest reports, Opus 4.7 at roughly 448 lines and GPT-5.5 at roughly 239 lines, stayed information-dense without unnecessary padding.
• The most common false-positive pattern involved overstating .env defaults as critical and treating mTLS as a universal fix. Several agents confused development defaults and internal trust assumptions with the genuinely exploitable issue: a browser-shipped credential trusted by the backend as service identity. Opus 4.6 also incorrectly overstated xml entity-resolution behavior.
• No evidence suggests cross-agent contamination. Reports contained no shared fabricated claims or verbatim reuse. Convergence around infra defaults, .env handling, the build script, and specific Envoy CORS line references appears independently derived from the same repository files.
• All agents appropriately avoided production probing and refrained from exposing raw secret values in their outputs.

Repo-scale security review results

The biggest thing this benchmark changed for me is that repo-scale AI-assisted security review is already operationally useful today. Even ordinary coding agents were often capable of:

• Traversing large repositories,
• Connecting frontend and backend paths,
• Reconstructing auth chains,
• Identifying blast radius,
• Surfacing remediation ideas,
• and finding secondary security issues worth human review.

Several models consistently reconstructed the full vulnerability chain without being pointed toward the correct files ahead of time. Even weaker outputs often surfaced useful secondary findings:

• cache retention issues,
• ingress hardening gaps,
• session handling problems,
• excessive logging exposure,
• auth inconsistencies.

The benchmark also reinforced that smaller models are no longer automatically disqualified from meaningful security review work.

• GPT-5.4 mini repeatedly identified the main issue correctly across all three runs.
• GPT-5 mini found the chain consistently as well, although it occasionally failed prioritization.

Capability is no longer confined to only the largest frontier systems.

The real failure mode was prioritization— Finding bugs is cheap now, knowing which one to fix first is not

The most important failure in these runs was prioritization. Many agents technically found the vulnerability chain while still failing the actual task. Claude Haiku 4.5 was the clearest example. Across all three runs it identified the vulnerable path. But across all three runs it buried the auth-boundary failure beneath safer, easier, more generic security commentary:

• JWT startup validation,
• internal gRPC hardening,
• committed .env defaults,
• generic secret-management recommendations.

The real issue was about browser-accessible code having access to a credential the backend accepted as trusted service identity. This is why I do not treat “found but buried” as a cosmetic failure mode.

A clean miss tells you the model did not get there. A buried hit is worse because it creates the illusion of competence while nudging reviewers toward lower-priority work.

This became one of the clearest lessons from the benchmark: finding vulnerabilities is increasingly cheap; understanding consequence hierarchy is not.

The contrast between models made this especially visible:

• GPT-5.4 mini consistently elevated the auth-boundary break as the primary issue.
• GPT-5 mini succeeded in 2 out of 3 runs.
• Claude Haiku consistently found the chain but repeatedly failed to recognize its importance.

Verbosity also turned out to be a surprisingly dangerous failure mode. Some reports looked thorough while actively making prioritization worse. Long reports filled with inflated severity ratings, generic hardening advice, and sprawling security commentary often obscured the real results.

What this changes about AI-assisted security workflows

The practical lesson from this benchmark is not “replace security engineers with agents.” It is almost the opposite. Repo-scale AI-assisted security review is becoming genuinely useful precisely because humans can now spend less time on repo traversal and more time on judgment. The workflow that increasingly makes sense looks something like this:

• Cheaper models perform broad reconnaissance,
• Multiple runs widen search coverage,
• Stronger models adjudicate findings,
• Humans evaluate consequence and remediation priority.

This benchmark also convinced me that future evaluations need to separate:

• “found the chain”
• “understood the consequence.”

Those are not the same capability. Most public AI security discourse still treats vulnerability detection as the headline milestone. I increasingly think the harder and more important capability is prioritization under uncertainty. The important shift is that repo-scale AI-assisted security review is already operationally useful with ordinary models and ordinary tooling.

The next phase of AI-assisted engineering will likely depend less on isolated models and more on end-to-end operational systems that can coordinate context, reasoning, prioritization, remediation, and governance together.