TechCrunch reports that AI companies have been tightening restrictions on the use of their models in cybersecurity scenarios in recent months. Restrictions originally designed to prevent abuse by malicious attackers are now impacting the productivity of legitimate defense teams and offensive security researchers. The crux of the controversy lies in the fact that the same set of capabilities can be used to both patch vulnerabilities and discover and exploit them, making it difficult for platforms to completely separate the two.
US model restrictions continue to tighten
The report mentions that the US government imposed export controls on Anthropic's Mythos and Fable models in June, at least in part, due to a report that claimed the security restrictions on these models could be bypassed and used to build or execute malicious cyberattacks.
Fable 5 subsequently reopened to the public on July 1st, while Mythos 5 only reopened to approved U.S. institutions and remains under government review. In addition to Anthropic, OpenAI also has a review program for cybersecurity researchers, allowing approved users to use models with fewer restrictions.
Researchers say it affects vulnerability verification
Several security researchers interviewed believe that current restrictions are beginning to hinder normal research processes. Chris Anley, chief scientist at NCC Group, stated that allowing models to attempt to exploit a flaw is a crucial step in determining whether it constitutes a real vulnerability. If the model directly refuses to respond, it becomes more difficult for defenders to determine whether the issue requires priority fixing.
He argues that requiring a model to "fix this code" inherently possesses both defensive and offensive attributes. This is because fix suggestions often expose critical weaknesses in the code, making it difficult for a platform to retain only defensive uses while completely eliminating offensive capabilities.
Security researcher Mark Dowd also criticized the practice, saying it's unsettling to have large AI companies unilaterally decide which security research is "safe." The report points out that Dowd has long been involved in zero-day vulnerability discovery and trading, and therefore he acknowledges that his stance may be biased professionally.
Some teams are shifting to local open-source models.
Some respondents indicated that when mainstream closed-source models cannot complete a task due to limitations, they would switch to open-source models without security restrictions. Paolo Stagno, CTO of CrowdFense, stated that his team uses cutting-edge models for reverse engineering, but prefers locally deployed open-source models for vulnerability discovery and exploit building.
His reasons included not only overly strict restrictions but also data security concerns. If sensitive vulnerability information is input into a cloud-based model, the relevant content could be leaked or incorporated into subsequent training processes. Locally running open-source models, on the other hand, do not require sending data to external platforms.
Another researcher from a smartphone component manufacturer said that because his company was not part of Anthropic's audit program, the tools were of little use in vulnerability discovery due to overly strict restrictions.
Concerns that researchers are being pushed to overseas models
Chris Thompson, CEO of cybersecurity firm RemoteThreat, stated that even within the review programs of Anthropic and OpenAI, the triggering methods for model restrictions are often inconsistent, with the same type of request potentially yielding different results each day. Researchers are thus forced to spend time repeatedly "negotiating" with the model, rather than focusing on vulnerability analysis itself.
He also stated that this situation is pushing responsible researchers towards locally run, uncensored Chinese open-source models, such as GLM. According to him, if US AI companies continue to tighten restrictions without expanding compliant access channels, defenders may lose their speed advantage in this AI-driven security race.
