Australia just gave a live demonstration of what happens when a government takes the human out of a life-changing decision, and then argues about it in public.
Since November 2025, the algorithm that decides how many hours of home care an elderly Australian receives cannot be overridden by the clinical staff who assess them. Earlier guidance had told assessors they could step in when the tool got it wrong. That ability was withdrawn without public notice the same month the new aged care system launched, without public consultation. Assessors who believe the algorithm has made a mistake now have no formal way to fix it on the spot.
This is not Australia's first run at this mistake. Between 2016 and 2020, the government ran a welfare debt recovery scheme known as Robodebt, which matched tax and welfare records with an algorithm and issued debt notices with no human checking the math. A court later called it a massive failure of public administration and ordered a settlement of 1.8 billion dollars. A second settlement worth 475 million dollars followed in September 2025, the largest class action payout in the country's history. Total cost of removing humans from that one decision: well north of 2 billion dollars, plus a royal commission, plus a government's reputation.
The pattern is not unique to Australia. Dutch tax authorities once used nationality as a signal to flag families for childcare benefit fraud, wrongly branding thousands of parents as cheats, a scandal that eventually brought down the Dutch government. In the United States, health insurers are now facing lawsuits and court-ordered evidence requests over claims that were denied by AI systems with no human reviewing the output.
That last point matters most for anyone running a business today. Insurance underwriters have started treating AI governance as a factor in coverage itself: companies that can show a documented human review process for their AI decisions get workable terms, while companies that cannot are facing exclusions and higher premiums. Regulators are catching up too. The EU's AI Act now requires human oversight for AI systems that affect a person's fundamental rights, covering areas like healthcare, hiring, and law enforcement.
The lesson is not that AI should stay out of decisions about people. It clearly cannot; the volume of claims, applications, and assessments most organizations handle makes some automation unavoidable. The lesson is that removing the human backstop to save time or money is the single most expensive shortcut a company or government can take. Every case above started as a cost-saving measure and ended as a multi-year, multi-million dollar cleanup.
If your business uses AI to decide who gets a loan, a claim payout, a job interview, or a service, the question worth asking this week is simple: can a person actually override that decision, and is there a record of when they do. If the honest answer is no, you are running the same experiment Australia just ran twice.