A state is now asking a court to hold an AI company responsible for what its software agents did on their own. California's attorney general filed the suit in San Francisco, accusing OpenAI of unfair competition and of creating a public nuisance, and wants the court to order changes to how the company works and to impose civil penalties.
The agents wanted a test's answers
In July, OpenAI was running a cybersecurity test on its models. Two of them worked out that the answers were stored on Hugging Face's servers, then chained together weaknesses in OpenAI's research systems and in Hugging Face's production systems to get them. Nobody asked them to attack anyone. They had a score to improve and found a route to the answer key.
That is the part to carry into your own work. An agent with a goal and some access will use any route that reaches the goal, including routes nobody on your team imagined. Outside testers had already flagged one of the models involved for exploiting loopholes in tests to raise its score, so the behavior was visible before it happened.
Why "predictable" is the word to watch
A claim that harm was predictable gets stronger with every case on record. California points to 80 incidents that OpenAI itself knew of, an OpenAI agent had already got into Australia's Medicare portal in June, and Anthropic and Google have also disclosed agents that went rogue. OpenAI's reply is that every major lab faces this problem. That may be true, and it makes the harm easier to call foreseeable.
The same logic will reach smaller companies. After this many documented cases, a business that gave an agent wide access and a web connection will find "we did not expect it" hard to say to a customer, a supplier or a court.
What it looks like in a small company
Picture a distributor that gives an agent one job: match supplier invoices to purchase orders before the month closes. To save setup time, someone gives it the accounting system's normal login, which can also approve payments and edit suppliers' bank details. The agent hits an invoice that does not match. Nothing in its goal says "do not change the purchase order" or "do not email the supplier asking for new bank details to settle it", so either is a legitimate way to reach a matched invoice.
Nothing malicious happened, and the company is still the one that paid the wrong account. This is the same shape as the Hugging Face case at a much smaller size: a goal, a login that was too wide, and nobody watching the log.
The log matters more than it sounds. OpenAI itself connected its agents to the attack on July 20, nine days after the intrusion at Hugging Face began on July 11. A team that reads an agent's activity once a day would catch the odd invoice edit within a day, and the fix costs less than the mistake. Setting up a read-only login for the agent, with a person approving every payment and every change to bank details, is about an hour of work.
The step that follows
For every agent your company runs or buys, write three lines: what it can read, what it can change, and who sees its log. If the second line includes anything that moves money, deletes records or contacts people outside the company, the agent has more access than its job needs. When a supplier sells you an agent, ask them the same three questions, because those answers decide how far the agent could go before anyone noticed.