Look at the rules banks, hospitals and security teams are writing for AI, and the same four turn up: a person reviews what the system flags, an override needs a supervisor's approval, decisions are logged and explained, and riskier cases get closer checks. Each one is sensible. The trouble is what each one tells the person who has to live by it. People read a rule as a statement about their job, and then they do the job the rule describes.
Each rule tells people something about their job
When the AI's answer comes first, it becomes the starting point. A person who used to ask "what is happening here?" now asks "do I have a reason to dispute this?" When the system flags nothing, the missing alert reads as proof that nothing is wrong. Employees already click approve after the AI decides, and a review rule written without care makes that worse. Less experienced staff slide fastest, because they have less feel of their own to set against the machine's.
The override rule says something different: judgment belongs to someone above you. A nurse who doubts a sepsis alert from Epic's system has to bring the doubt to a physician, which means interrupting someone with more status. The person closest to the patient has the best information, and the rule sends the decision to people further away.
The logging rule teaches people to reason backward. In money-laundering checks, 90% to 95% of alerts are false positives, and each one still has to be investigated and written up. After a few hundred of these, an analyst learns to write what an auditor will accept. A hunch they cannot put into words, which is often how experience shows up, has nowhere to go. Risk tiers add their own message: decide in advance which cases matter. People then relax on the cases labelled routine, which is where early warnings can first show up. It is the same drift that happens when heavy AI use can wear down professional skills, only here the rule is doing it.
What a small change looks like
Take an accounts-payable clerk at a mid-sized distributor. An AI flags suspicious invoices, such as a duplicate or a supplier whose bank details changed, and the clerk's rule is to review each flag and sign off. Within weeks the clerk is checking flags, not invoices, and anything unflagged goes through at speed.
Now change the first step on a random one invoice in ten. Before the flag is visible, the clerk types a line: "same supplier, same amount, fine" or "bank details differ from last month, looks off". Then the clerk sees what the AI said. At 200 invoices a day that is 20 short lines, about three minutes. The clerk starts from their own reading again, and every disagreement between the line and the flag is on record.
Two more changes follow the same logic. When the clerk disagrees with the AI, they ask a colleague for a second look instead of sending the case up for approval, so doubt costs a conversation and not a favour from a boss. And the case form gets a field for the basis of a decision, with choices like "followed the rule", "matches a pattern I have seen before" and "something feels wrong but I cannot say why". Experience then counts as a valid reason in the record.
Disagreement is the number to watch
A team that never disagrees with its AI is not necessarily safe. It may be a team that stopped looking. That gives a company a check no test of the model can give: how often a person's first read differs from the system's, and whether people can say why when they agree. If the rate is near zero, the oversight rule is producing signatures, not a second pair of eyes.
Pick one process where people approve what an AI flagged, and change its first step so they write their own read before they see the AI's.