Claude Mythos Preview cyber evaluation by UK AISI

In plain words

The UK AI Security Institute (AISI) tested whether Anthropic's Claude Mythos Preview can carry out computer attacks largely on its own. In a simulated company network with 32 steps, from first reconnaissance to full network takeover, it became the first model to finish the whole job, in 3 of its 10 attempts. AISI estimates that a human would need 20 hours for the same range. The simulated networks had no active defenders, so AISI says it cannot tell how the model would fare against a well-defended system. The result matters because it shows what an AI model can do when someone directs it and gives it network access.

What was tested

AISI evaluated Claude Mythos Preview, which Anthropic announced on 7th April, to assess its cybersecurity capabilities. The institute says it has tracked AI cyber capabilities since 2023 and has built progressively harder evaluations to keep pace, moving from chat-based probing to capture-the-flag challenges and then to multi-step cyber-attack simulations.

The first kind of test was the capture-the-flag (CTF) challenge. In these, a model must identify and exploit weaknesses in a target system to retrieve hidden flags. AISI points out that such challenges test specific skills in isolation. Real-world cyber-attacks, it says, require chaining dozens of steps together across multiple hosts and network segments, sustained operations that take human experts many hours, days or weeks to complete.

The second kind was a cyber range, a simulated network that the model attacks across many steps. AISI built one called "The Last Ones" (TLO), a 32-step corporate network attack simulation spanning initial reconnaissance through to full network takeover. AISI estimates it requires humans 20 hours to complete. The institute also reports on a range called Cooling Tower, which focuses on operational technology (OT), the equipment-control side of an industrial network.

A good result on the CTF suite is a high success rate at each difficulty level. A good result on TLO is completing more of the 32 steps, and at best all of them.

How the test works

AISI describes its CTF chart as showing each model's average success rate at a given difficulty level, with Mythos Preview compared against other models. For expert-level tasks, the institute states that no model could complete them before April 2025.

The range evaluations were controlled. Mythos Preview was explicitly directed to attack and given network access to do so. AISI says it ran the cyber ranges with a 100M token budget. Mythos Preview made 10 attempts at TLO, and AISI reports both the number of attempts that solved the range and the average number of steps completed across all attempts.

The only other model the report names with a figure is Claude Opus 4.6, which AISI identifies as the next best performer on TLO.

AISI also names features that make its ranges easier than real environments. They lack security features that are often present, such as active defenders and defensive tooling. There are also no penalties for the model for undertaking actions that would trigger security alerts.

Results

On the CTF suite, AISI reports continued improvement. On expert-level tasks, the institute says Mythos Preview "succeeds 73% of the time."⁠¹ No model could complete tasks at that level before April 2025.

The larger finding concerns the range. AISI reports that Claude Mythos Preview is the first model to solve TLO from start to finish, and that it did so in 3 out of its 10 attempts. Across all of its attempts, the model completed an average of 22 out of 32 steps. AISI says Claude Opus 4.6, the next best performing model, completed an average of 16 steps. AISI summarises the overall picture as continued improvement on CTF challenges and significant improvement on multi-step cyber-attack simulations. It describes Mythos Preview as a step up over previous frontier models.

AISI also reports a limitation. Mythos Preview could not complete the Cooling Tower range. The institute cautions that this does not necessarily show the model is bad at executing attacks in OT environments, because the model got stuck on IT sections of that range.

On computing resources, AISI reports that Mythos Preview's performance continues to scale up to the 100M token budget it used. The institute says it expects performance improvements would continue beyond that limit. That part is an expectation, because the runs stopped at that budget.

What the results do and do not show

What AISI measured: a 73% success rate on expert-level CTF tasks, 3 full solves of TLO in 10 attempts, and average step counts of 22 for Mythos Preview and 16 for Claude Opus 4.6. The report states these as results of its own controlled evaluations.

What AISI says the results suggest: Mythos Preview's success on one cyber range indicates that it is at least capable of autonomously attacking small, weakly defended and vulnerable enterprise systems where access to a network has been gained.

What AISI says it cannot conclude: its ranges have important differences from real-world environments that make them easier targets. In its words, "we cannot say for sure whether Mythos Preview would be able to attack well-defended systems."⁠¹ The institute also says that the Cooling Tower failure does not settle how the model handles OT environments.

TOSWO Research's reading: the figures describe what a directed model with network access can do against an undefended simulated network. They are not a measure of how often the model would succeed against a monitored network, and the report does not offer one. The TLO result also rests on 10 attempts by Mythos Preview, and the report gives no spread or error range for the averages, so small differences in step counts should not be over-read. The gap between 22 and 16 steps is a comparison of two reported averages, and the report does not say how many attempts Opus 4.6 made.

AISI itself says that, as capabilities improve, evaluation environments that lack defences will no longer be challenging enough to discriminate between the most cyber-capable models or assess trends.

Why it matters

AISI draws its own conclusion for organisations. Its testing shows that Mythos Preview can exploit systems with weak security posture, and it says more models with these capabilities will likely be developed. The institute points to cybersecurity basics: regular application of security updates, robust access controls, security configuration and comprehensive logging. It refers readers to the Cyber Essentials scheme run by its colleagues at the National Cyber Security Centre (NCSC). AISI also says investment now in cyber defence is vital, and notes that these capabilities are dual use: they pose security challenges, but they can also help deliver improvements in defence.

AISI also states its next steps. Its future work will evaluate capabilities using ranges that simulate hardened and defended environments, including active monitoring, endpoint detection and real-time incident response. It will also track how AI-enabled vulnerability discovery and penetration testing campaigns perform on real-world systems.

TOSWO Research's reading: if an attacker can direct a model and has reached a poorly defended network, the report indicates the model can chain many steps without human help in a simulated setting. Organisations with weak basic security match the profile AISI describes. Readers who want to compare this with other agent tests can browse the simulations section, and the page on TOSWO's method explains how the site approaches measuring AI risk.

Questions readers ask

Did Claude Mythos Preview complete the full network attack simulation?

AISI reports that it did, in 3 out of its 10 attempts, making it the first model to solve The Last Ones range from start to finish. Across all its attempts it completed an average of 22 out of 32 steps.

How does Mythos Preview compare with other models in this test?

AISI names Claude Opus 4.6 as the next best performer on the range, with an average of 16 steps completed. For the CTF suite, AISI reports that Mythos Preview succeeds 73% of the time on expert-level tasks.

Does this mean the model can hack well-defended companies?

AISI says it cannot say for sure. Its ranges had no active defenders or defensive tooling, and the model faced no penalty for triggering security alerts. The institute plans future work on ranges that simulate hardened and defended environments.

What did AISI recommend after the test?

AISI recommends cybersecurity basics: regular security updates, robust access controls, security configuration and comprehensive logging. It points to the NCSC Cyber Essentials scheme and says investment in cyber defence is vital.