A team building an AI model to predict breast cancer risk from mammograms found, partway through training, that the model could name the hospital behind each scan, though nothing in the data named it. It had picked up tiny differences between scanners and was using them as a shortcut. The team engineered the shortcut out, and the finished model became the first AI the FDA authorized to predict five-year breast cancer risk from a mammogram alone. The lesson reaches well beyond mammograms.
## A model takes whatever route pays Ordinary software does what its rules say, and you can read the rules. An AI model has no rules to read. It adjusts itself until it gets the training examples right, and it does not care how. If one hospital sends more high-risk patients than another, then "which hospital is this?" helps it guess the answer, and the scanner marks give that away for free. Nobody asked for the shortcut. It was simply the easiest way to score well.
That is why a model can look excellent before it is used and disappointing afterward. In 2020, researchers examined AI systems that detected COVID-19 on chest x-rays. The systems had been trained on images pooled from different hospitals, and they relied on marks tied to where the image came from rather than on signs of disease. They looked accurate and then did worse when tested at new hospitals.
## The same trap appears in ordinary offices Take a finance team at a distributor that is offered AI to read supplier invoices and assign each one an expense code. The vendor shows impressive accuracy, measured on invoices from other companies. Maybe the tool learned that a certain layout from a certain large supplier means "freight". Your own mix is different: phone photos of paper delivery notes, a handful of small suppliers, an invoice that lists three kinds of cost on one page. Accuracy can be high overall and poor on exactly the invoices that take your staff the longest.
You can find this out cheaply. Pull 300 invoices from last quarter that your team already coded. Hide the codes, run the tool, and compare. Then split the results by supplier type, because an average hides the weak spot. That is about half a day for one person, and it tells you more than any demo will.
The same test works for a recruiter trying a résumé screener on last year's applicants whose outcomes are known, or a warehouse manager trying a demand forecast on last winter's orders. And it has to be repeated when something changes: a new scanner, a new supplier mix, a new customer base. A shortcut that was harmless in the old setting can become the main thing the model relies on in the new one.
## Evidence from another place is a starting point Hospitals show what happens when this step is skipped. Epic's sepsis alert was in use at hundreds of hospitals in 2021, when University of Michigan researchers checked it on their own patients. The alert missed 67% of the patients who developed sepsis, while it raised alerts for 18% of all patients. A warning that misses most cases and still interrupts nurses all day is worse than useless in one specific way: staff may feel covered by a safety net that is not there.
This is the real difference from buying software. With a spreadsheet or a payroll system, a demo in someone else's office tells you most of what you need. With AI, how well it works is a fact about a particular set of data, and the only data you can be sure of is your own. So the question to put to any vendor is where the tool has been tested on data that was not used to build it, and whether it can be tested on yours before you sign. A vendor that welcomes that test is showing you something about the product. One that resists it is showing you something too.