Most AI test scores describe other people's work, so a test of 20 of your own tasks tells you more

An AI test, called a benchmark, is a fixed set of tasks with known good answers, and model makers quote the scores when they release a model. When 29 widely used benchmarks were compared with the 18,797 tasks that the US Department of Labor lists for 923 occupations, fewer than one task in twenty had real coverage. The tests favour tasks from the occupations with the largest total wages.

So a model's score usually describes someone else's job. For a dispatcher or a hospital billing clerk, it says nothing either way: missing coverage does not mean AI fails at the work, only that nobody has checked.

A team can check for itself. For example, the head of customer service at a freight forwarder collects 20 real requests from last month, such as a late container, a damage claim and a rate question. Next to each goes the reply that the team actually sent. Two assistants answer all 20, and an experienced team member marks each answer as ready to send, needing edits, or wrong. Building the set takes about a day, and running it again when a new model comes out takes about an hour.

Nobody outside a company will ever test the tasks that fill most of its working hours. A team with its own 20 tasks will know which work it can hand over while its rivals are still reading scores about other people's jobs.

The Kraft Group owns the New England Patriots. Its technology chief, Michael Israel, wants each new AI use case to answer how someone will act on it. His examples include an alert telling stadium staff that a bathroom line keeps growing during a concert.

Hospitals show what happens without that answer. Bedside monitors sound so often for things that need no action that nurses stop reacting quickly. Every alert that nobody acts on teaches people to ignore the next one. So the bathroom alert is finished only when a named supervisor can send help before the line clears.

OpenCode, an AI coding tool, ran a web dashboard that stored teams' OpenAI, Anthropic and Google keys. Someone broke into an old version of it and used the keys to send huge numbers of requests, which the key owners paid for.

A provider charges a key's owner for every request until someone stops it, and the usage alert or the monthly invoice reaches that person after the money is gone. An agent stuck repeating one step all night spends the same way.

In Europe, mobile data used abroad produced huge phone bills until 2010, when the EU made a €50 cut-off the default: the phone stopped using data until its owner agreed to pay more. Each AI workflow needs the same default: its own key, a named owner and a limit at the provider that stops the key.

Muse, Meta's AI agent, turned each shared chat into a page that anyone with its web address could open without signing in, and it put the chat's title in that address. When a recipient opened the page, their computer looked up the address in the internet's directory, and services that record such lookups kept it in a searchable history. Pages found that way included a small business loan application.

Meta has paused sharing and the pages no longer open, but the titles stay in that history. A title usually names what was private: the client, the loan, the quarter.