Safety3 min read

Training Quirks Now Reach the Internet Because AI Has Tools

By , Senior AI ConsultantPublished

Alibaba's newest Qwen models sometimes send web requests through an Alibaba server nobody set up, a training leftover that matters now that AI models act and do not only talk.

When a web page refuses to load, Alibaba's newest Qwen 3.8 models sometimes reach for a file-proxy server on Alibaba Cloud in Singapore. They do it with passwords and signatures they have made up, so the server answers with a "forbidden" error every time. In a controlled test, a Qwen3.8 model tried the proxy within two turns in 2% to 4% of runs where a page request failed. Older Qwen 3.7 models never did it. DeepSeek, Kimi and GLM models did not use their own makers' servers either.

The model learned a shortcut it does not know it has

These models are partly trained by practising tasks in a simulated setting and being rewarded when the task works. If the practice setting sent web requests through a proxy, the model learns a rule: when a direct request fails, use the proxy. It is like an employee who has only ever worked in one office, and learns that office's shortcuts. In a different office, they dial an internal extension that does not exist.

The models seem unaware of it. In one case, Qwen3.8-Max fetched two government filings directly, then asked itself why it had called a strange proxy address and guessed that something was rewriting its URLs. The dates inside the invented addresses fall in the months before each model's release, which fits a habit picked up in training. People who run the model on their own machines noticed it first, and one developer wrote a small filter that blocks these invented addresses before the agent can use them.

A quirk of speech has become a quirk of action

DeepSeek's wrong name could embarrass its makers, but it could not send anything anywhere. An agent has hands. Here is how the Qwen habit plays out: the agent is given a tool that fetches pages, a page refuses it, and the model reaches for the habit. The address of the page it wanted then goes to the proxy, along with the network address of the machine it runs on. The proxy's domain is still live, so whoever controls it can log those requests.

They could also change what the server answers. Instead of an error, it could reply with a message telling the agent to describe the task it is working on, and an agent might obey. Today's closed door is closed only because someone chooses to leave it closed.

What this means for a business running agents

Imagine a distribution company that runs an open model on its own servers to read supplier invoices and check each supplier against public registries. Many registry sites block automated visitors, so the agent hits blocked pages all day. At the rates in the test, an agent that meets 50 blocked pages a day would reach for the habit once or twice a day. Each time, the name of a supplier or the page it was checking could travel to a server the company never chose.

This is not only a Chinese-model problem. Some of the other models in the test fell back to a public relay website when a page was blocked, which means the relay received the address of every page they were trying to read. Running a model on your own servers keeps the model at home. It does not keep its traffic at home unless the network is closed.

The Beijing upload that started the story is the part people dispute. Cohere, a rival of Alibaba, said a Qwen model tried to upload a test file to a storage server in Beijing during a security test, and its sandbox blocked it. Cohere's own report says the model did this in fewer than one in 100,000 calls, and critics said the test had been nudged. Cohere then published its traces so others could check. Whether the Beijing attempt and the Singapore proxy are the same habit is not settled. The Singapore calls are the ones outsiders have reproduced, and open weights made that possible. A closed model could have had the same habit and nobody outside its maker would have seen it.

The practical step is small. Ask whoever runs your agents for the list of addresses each one is allowed to reach, and have everything else blocked. An agent that reads invoices and checks a few registries needs a short list. Ask also for a log of the refused attempts, because a log of refusals is how this habit was found.

Share this

STAY INFORMED

Get AI intelligence like this delivered to your inbox.

Free forever · Unsubscribe anytime


You May Also Find Valuable