Open weight AI models on cyber tasks, UK AISI gap test

In plain words

The UK AI Security Institute (AISI) asked how far the best freely downloadable AI models trail the best closed, company-controlled models at hacking-related skills. It tested two recent open models, GLM-5.2 and DeepSeek V4-Pro, and reports that they perform similarly to closed models released 4 to 7 months before them. For open models released from January to September 2025, AISI measured a gap of 6 to 10 months. AISI treats the gap as preparation time: a window for defenders to act before today's frontier cyber capabilities might become available without the same safeguards.

What was tested

AISI separates "open weight" models from "closed weight" ones. Open weight models have parameters that anyone can download, run and modify. Closed weight models are reachable only through interfaces the developer controls. AISI says it has tracked the cyber capabilities of frontier models since 2023 and that the most capable models have consistently been closed, with open models trailing.

This report is AISI's first public analysis of that trailing distance. It chose two open models because they were candidates to lead open weight cyber capability when they were released: GLM-5.2 and DeepSeek V4-Pro. AISI describes GLM-5.2 (June 2026) as the most cyber-capable open weight model at the time of testing. The closed models named for comparison include Opus 4.6 (Feb 2026), Opus 4.5 (Nov 2025), GPT-5.3-Codex and Sonnet 4.5.

Two kinds of ability were measured. The first is narrow cyber tasks, which probe specific skills such as vulnerability research and exploitation, reverse engineering, web exploitation and cryptography. The second is autonomous attack ability on cyber ranges, where a model must plan and carry out a long, multi-step attack on a simulated network.

How the test works

The narrow cyber tasks are graded across four difficulty levels, from "technical non-expert" (for example a data analyst with limited cyber knowledge) to "expert" (tasks that typically need deep cybersecurity knowledge). AISI notes in a footnote that it used a 70-task subset of its full 96-task suite to enable historical comparisons. The subset has 18 technical non-expert tasks, 25 apprentice tasks, 19 practitioner tasks and 8 expert tasks. A second footnote says comparator models were unchanged when evaluating at 50M tokens per task with the full 96-task suite.

The cyber ranges are expert-built simulated networks of hosts, services and vulnerabilities, arranged as sequential attack chains that start at the point of initial network access. AISI points out that the ranges lack features of well-defended real environments, namely active defenders, defensive tooling and alert penalties. The range featured in the report, "The Last Ones" (TLO), is a 32-step corporate network attack across 4 subnets and approximately 20 hosts. AISI estimates a human expert would need roughly 20 hours to complete it. Results on this range are shown as model trajectories, meaning how far through the steps each model gets.

AISI also compared costs at advertised first-party prices. It says it did not use first-party providers for the open weight models it tested, so real compute costs may vary. It says it did not pursue specific elicitation or optimisations that could have improved performance, so its setup likely slightly underestimates what the open models can do at most.

Results

On the narrow cyber tasks, AISI reports that GLM-5.2 performs comparably to the most cyber-capable models released 4 months before it, Opus 4.6 and GPT-5.3-Codex, and that this holds across all four difficulty levels. DeepSeek V4-Pro is comparable to Opus 4.5, which was released 5 months before it. AISI says both gaps are narrower than in its internal 2025 evaluations, when open weight models lagged the frontier by 6 to 10 months.

On the cyber range TLO, the gap is larger. GLM-5.2 reaches as far as Opus 4.5, a model released less than 7 months before it. DeepSeek V4-Pro falls below Sonnet 4.5, which AISI calls a sub-cyber-frontier model released 7 months before it. AISI says these results are broadly consistent across its other ranges. It adds that GLM-5.2 reached step 7 with marginally fewer tokens than any other model on average, and that it tracked Opus 4.6's trajectory to step 11 before stalling. Overall, AISI states that GLM-5.2 performs similarly to Opus 4.6 on its narrow cyber tasks and Opus 4.5 on its longer-horizon cyber ranges, meaning it trails the frontier by 4 to 7 months.

AISI says its evaluations of recent open weight models were largely unimpeded by safeguards. DeepSeek V4-Pro occasionally refused narrow cyber tasks, mainly in reverse engineering, and AISI circumvented this with a small number of repeat attempts at the refused tasks.

Cost differed sharply. AISI reports that a 100M-token cyber range run cost roughly $85 for both Opus 4.5 and Opus 4.6, against an estimated $46 for GLM-5.2 and $1.19 for DeepSeek V4-Pro. On tasks that both models in a pair solved with 100% reliability, AISI reports that Opus 4.6 cost $15.17 per task versus $6.12 for GLM-5.2, and Opus 4.5 cost $12.50 per task versus $0.28 for DeepSeek V4-Pro. The two Opus models share one figure on range runs, so they cannot be ranked against each other there.

1. Opus 4.5$85
2. Opus 4.6$85
3. GLM-5.2$46
4. DeepSeek V4-Pro$1.19

0255075100

The open weight models are priced at current advertised rates, and AISI says real compute costs may vary.

Cost of a 100M-token cyber range run: the figures as the tester reports them

Entry	Cost of a 100M-token cyber range run
Opus 4.5	$85
Opus 4.6	$85
GLM-5.2	$46
DeepSeek V4-Pro	$1.19

What the results do and do not show

AISI states several limits itself. The range comparison is weaker evidence than the narrow task comparison, because it comes from a smaller set of ranges than the larger narrow task suite. Trajectories alone also do not show whether a stall comes from insufficient cyber capability at a given step or from insufficient agentic capability to sustain long-horizon planning and execution. The ranges lack active defenders, so success on them is not the same as success against a defended network. The post tests cyber capability only, and AISI says inferences to other capabilities cannot be drawn. The cost figures use advertised prices, and AISI did not use first-party providers for the open models.

AISI also says the narrowed gap is not predictive. It describes the result as "not predictive of whether future open weight models will replicate the more recent jumps delivered by Mythos Preview and GPT-5.5"⁠¹. Those two closed models showed, in April 2026, some of the largest jumps in AI cyber capability AISI has observed since testing began. The 4 to 7 month figure therefore describes the models tested against earlier closed models, not the newest closed frontier. For AISI's own evaluation of one of those closed models, see the Claude Mythos Preview cyber evaluation by UK AISI.

TOSWO Research's reading: the two measures give different numbers because they test different things, and AISI itself treats the range result as the weaker one. A reader may therefore treat 4 to 7 months as a band rather than a precise lag.

Why it matters

AISI concludes that recent open weight models lag frontier closed cyber capabilities by 4 to 7 months, and that this implies cyber defenders have a short window to prepare before today's frontier cyber capabilities may become accessible without the same safeguards. It notes that the National Cyber Security Centre has already encouraged organisations to invest in cybersecurity baselines and to leverage AI-enhanced defences, and says its findings underscore that message. AISI also stresses that open weight models bring real benefits, such as private hosting, adaptation to specific tasks and support for open collaboration, and that the challenge is balancing those benefits with security. It says no current approach guarantees safety, but several strategies in combination could meaningfully reduce risk.

TOSWO Research's reading: the cost figures show that the open models tested were priced well below the earlier closed models they were compared with, at advertised prices. That is a comparison of AISI's pricing numbers, not a finding about how widely the models are used. AISI says it will continue to evaluate leading open weight models, including Kimi K3 once its weights are publicly released, which it says was announced for the end of July. Readers who want to see how such tests are built can look at TOSWO's page on how AI agents perform in tests and its description of how TOSWO measures AI risk.

Questions readers ask

What is an open weight AI model?

AISI defines open weight models as those whose parameters anyone can download, run and modify. Closed weight models are accessible only through interfaces the developer controls. AISI says closed developers can use that control to detect and disrupt misuse and withdraw models, and that these options are lost permanently once open weights are released.

How far behind are open models on cyber tasks?

AISI reports that GLM-5.2 and DeepSeek V4-Pro perform similarly to closed frontier models released 4 to 7 months before them. In its 2025 internal evaluations the gap was 6 to 10 months. AISI says the result does not predict whether future open models will match the latest closed jumps.

Were the open models stopped by safety refusals?

AISI says its evaluations were largely unimpeded by safeguards. DeepSeek V4-Pro occasionally refused narrow cyber tasks, mainly in reverse engineering. AISI circumvented this with a small number of repeat attempts at the refused tasks.

Does this test show the models can attack real networks?

No. AISI says its cyber ranges are simulated networks that lack active defenders, defensive tooling and alert penalties. It also treats the range evidence as weaker than the narrow task evidence because it is drawn from a smaller set of ranges.