Claude is competitive with humans in (some) cyber competitions

Throughout 2025, we have been quietly entering Claude in cybersecurity competitions designed primarily for humans. Now, we want to share what we have learned. In many of these competitions Claude did pretty well, often placing in the top 25% of competitors. However, it lagged behind the best human teams at the toughest challenges.

Our experience testing Claude in cyber competitions highlights the potential for AI to alter the offense-defense balance by making it easier for attackers to automate the exploitation of basic vulnerabilities. More research and development into AI-enabled cyber defense and resilience is needed to counter this development.

AI is poised to transform the domain of cybersecurity. Anthropic’s Safeguards team recently identified and banned a user with limited coding abilities leveraging Claude to develop malware. Research suggests that this lowering of the bar for expertise needed to pose a threat, combined with the falling costs of large language models (LLMs), presages a dramatic shift in the economics of cyberattacks. [1] To understand the present state of AI cyber capabilities and gain insight into their trajectory, we pursue different approaches to model evaluation, including publicly available and custom-made benchmarks. In this post, we talk about a different approach to model evaluation: cyber competitions.

Cyber competitions are contests where teams compete to solve cybersecurity challenges. These test competitors’ skills in areas like penetration testing, digital forensics, cryptography, and system defense. Examples include capture the flag (CTF) events like PicoCTF and AI vs Human CTF Challenge where participants solve puzzle-based challenges, as well as Collegiate Cyber Defense Competition (CCDC) where teams defend vulnerable networks against live attackers. These competitions range from beginner-friendly contests for high school students to expert-level events with large cash prizes for top finishers.

We have been entering Claude into these competitions because they provide several advantages for stress-testing the cyber capabilities of frontier AI models:

When Claude is able to solve a cyber challenge, it is as fast or faster than elite human teams. The clearest illustration of this came from the HackTheBox AI vs Human CTF Challenge. At the time the competition started, the Anthropic researcher responsible for launching Claude was busy moving into a new apartment. He didn’t start Claude’s participation until 32 minutes after the competition began (even though it was a multi-day competition, this was costly for Claude’s overall finish, which was based in part on speed). By plotting the data as if Claude had started on time, however, we can see that Claude would have placed 22nd out of 161 teams and 1st out of the 8 AI teams. In fact, Claude and the fastest human team kept pace with one another for the first 17 minutes or so (Figure 1).