It may sound like a sci-fi plot, but an artificial intelligence algorithm recently played a major part in the downfall of an elected government. 

In January 2021, the government of the Netherlands decided to step down. The scandal that set their resignation in motion began in 2012, with the introduction of a self-learning algorithm to seek signs of benefits fraud. The tax authority believed that the algorithm would flag up potential fraud cases, which humans could then assess to weed out any errors.

What played out, as IEEE Spectrum reports, was that the algorithm began to develop patterns of bias, disproportionately labeling those from lower-income backgrounds, immigrants, and ethnic minorities as having committed fraud. The humans tasked with vetting the cases – perhaps too trusting of the algorithm – accepted many of the false flags, and demanded that money be returned.

The consequences of the new system (and the humans in government who trusted it) cannot be overstated. Tens of thousands of people were left in poverty, while some died by suicide following giant tax bills – which sometimes went into six figures.

When Chermaine Leysner received a bill totaling €100,000, she believed that the government had made a mistake in asking her to pay back the child allowance she had received since 2008. 

“I was working like crazy so I could still do something for my children like give them some nice things to eat or buy candy," Leysner told Politico. "But I had times that my little boy had to go to school with a hole in his shoe."

Following the stress from the bill, and other personal circumstances, she separated from her children's father. She was far from alone.

The Ministry of Justice and Security Statistics commissioned an approximation of the number of children who had been placed in care following a false fraud flag. They found that more than 1,100 children had been separated from their families, placed in care, and kept from their families for years as a result of the scheme.

Around 70,000 children were affected by the bills, which were wrongly given to around 30,000 parents. The figure could be far higher, as the estimate was based on data after 2015.

Compensation of at least €30,000 is to be paid to victims of the algorithm and the humans that enabled its false assumptions, but it was not enough to save the government from the scandal, known in the country as kinderopvangtoeslagaffaire. 

"Innocent people have been criminalised and their lives ruined," Prime Minister Mark Rutte told reporters as his government took the unanimous decision to step down in January as a result of the scandal. "The buck stops here."