Safety3 min read

Cheap AI Routing Can Hand Private Prompts to Strangers

By , Senior AI ConsultantPublished

Secrets planted in AI prompts were stolen at three providers, and a prompt marked do not store still had its secret link shared in a Chinese company chat 24 days later.

Passwords, access keys and private links hidden in about two million AI prompts were stolen at three AI hosting providers, and sellers on a discount marketplace were reading what their customers sent. The model was not the weak point. The weak point was the chain of middlemen between an AI tool and the model, which most businesses never look at.

A prompt passes through more hands than the tool shows

Many AI apps do not call Anthropic or OpenAI directly. They call a router, which sends each request to whichever model is cheapest or best suited, and that router may hand it to another one. Every step can read the whole prompt and the reply, and the vendor that sold you the app usually chose this setup to keep its costs down.

In open marketplaces, the seller with the lowest price gets the request. A seller can undercut everyone by pooling consumer subscriptions or using startup credits, and a seller who wants data only has to list a popular model cheaply, switch on logging and wait. The ones HD Moore caught were individuals on home internet. About a day and a half after one prompt went through a seller, someone opened its two secret links in a desktop browser and started guessing usernames. In another case a person in France took passwords out of two prompts and typed root logins by hand. The same model sold by a different seller was never touched, so the risk follows the seller, not the marketplace.

The privacy setting covers the first step only

The strictest privacy setting is called Zero Data Retention: the prompt and the reply are not stored. A router is supposed to pass those requests only to providers that honour the setting, and often they did not. One Chinese AI lab's systems fetched the secret links from more than half of the prompts sent to its models, including about two-thirds of those marked as not to be kept. US providers that resell the lab's models set off the same scraper, so a customer who chose a US provider to keep data in the US did not get that.

The three breaches show the same limit. In two of them, some of the stolen requests carried the do-not-store flag. In the largest, the attacker copied requests at the provider's front door, even ones rejected before any model ran. A flag cannot help when the attacker is already inside, and none of the three providers knew until they were told.

The old leak needed a mistake; this one does not

In 2023, a security company planted AWS keys in public code repositories and watched them get used within two minutes. That was a leak caused by a person's slip, one developer pushing the wrong file to the wrong place. Thieves have been fast for years.

What is different here is that nobody slips. An AI agent puts whatever it can read into its prompt, so if it can read a settings file with a database password, the password goes out with every request, through every hop. A small IT team whose agent handles a few thousand requests a day is handing that password to each router and provider on the path thousands of times a day. The thieves in this case acted like thieves with a stolen key always do: they tested the bait keys, in some cases within hours, then listed storage buckets, searched for stored secrets and tried to add their own users so they could come back.

Where the prompt goes is a decision

None of this is a reason to stop using AI. It means the route is a choice, and if nobody makes it, the lowest price makes it. Ask the vendor of any AI tool you pay for which providers handle your requests and whether a router picks them. Keep passwords and keys out of anything an agent can read. Treat a privacy flag as covering the first hop only.

You can also find out for yourself. Create a fake AWS key with a free canary token service, put it in a file your agent reads, and set it to email you when someone uses it. If the email arrives, someone outside your company has read your prompts, and you have learned it before a real key was at risk.

Share this

STAY INFORMED

Get AI intelligence like this delivered to your inbox.

Free forever · Unsubscribe anytime


You May Also Find Valuable