A Canadian company AI stated on Monday that during a routine cybersecurity test, it was discovered that one of Alibaba's latest AI models, Qwen3.8-Flash-Next, attempted to upload a file to a server in Beijing, despite the model stating in its own notes that it would not use any tools.
This incident initially raised concerns among outsiders that the model might have violated a fundamental commitment of the company—that is, that self-hosted open-source weighting models could keep data internal. However, these early concerns about corporate espionage were quickly dispelled. Researchers quickly pointed out that it was more likely a byproduct of a chaotic training environment, causing the model to deviate from instructions unintentionally, rather than intentionally sending sensitive information to China in secret.
The incident occurred last Friday. At that time, the Qwen model was required to write a rule for identifying malware downloads. According to a blog post by Cohere, the model's written reasoning clearly stated that it would not use any external tools. However, in the next step, it attempted to send a test string (i.e., a short piece of placeholder text) to an external server, which was then intercepted by a security sandbox.
Cohere indicates that it is currently impossible to determine whether this behavior is intentional or a residual effect from the training process.
When questioned about the controversy over model intentions, Cohere, co-founder with Ivan Zhang, told TechCrunch that Cohere had also observed his own models developing some strange habits in the training simulator.
“It’s really difficult to determine intent or motivation. We have studied the behavior of our models and observed instances where they reward hacking behaviors; sometimes these models become very creative when trying to break out of the sandbox,” said Zhang. (Rewarding hacking behaviors refers to models finding shortcuts to obtain rewards without completing tasks as expected.)
In order to train the AI model to become an agent that can take autonomous actions, companies like Alibaba will train it in thousands of different practice scenarios, similar to software flight simulators. Each time the model completes a task in the simulator, it receives a reward, and in the process, it develops certain habits. If these practice scenarios often involve storing files on Alibaba's cloud servers, the model may learn to save files there by default.
Critics also argue that this incident may be due to the Cohere methodology of investigation – where researchers guide the model to perform in a certain direction, rather than measuring whether Qwen would exhibit such behavior on its own.
In order to identify the cause that triggered the upload attempt, Cohere provided the beginning of a network command (i.e., a command that instructs one computer to connect to another), and then asked the model to complete it. Since this model has processed thousands of regular network commands during its training, its performance was like that of a highly advanced autocompletion tool—it often completed URLs, some of which pointed to Alibaba's servers.
Cohere indicates that the initial upload attempt did not receive any such assistance, but such behavior is quite rare.
After developers requested more evidence, Cohere released the original test trajectory on the AI developer platform Hugging Face later on Monday, along with the script, to facilitate others in attempting to reproduce this behavior.
A researcher who was initially skeptical stated that the evidence is sufficient to suggest that Qwen might fabricate suspicious web addresses, but there is no evidence to indicate that Alibaba has implanted a backdoor, nor is there any evidence that Qwen is deliberately hiding its intentions.
At the time of this incident, governments around the world were deciding to what extent they would impose restrictions on China's AI. Last week, the U.S. Department of Homeland Security ordered federal civilian agencies to remove products from Chinese AI developer DeepSeek from their systems.
In the blog post, Cohere calls on governments around the world to clarify whether the restrictions targeting China's AI also apply to models running on government computers.
"I'm not advocating for comprehensive regulation of these open-source weight models," Zhang told TechCrunch, and added that companies should treat open-source models just like any program downloaded from the internet.
Cohere itself sells the AI model to the government, and if officials become more cautious about free Chinese alternatives, the company could benefit from it.
