This afternoon the MAA-AMS-SIAM Gerald and Judith Porter Public Lecture was given by Cathy O’Neil, “Big data, inequality, and democracy.”
O’Neil started by giving on overview of what she calls a Weapon of Math Destruction. They are “creepy algorithms,” O’Neil says, ones that are used by large groups of people to make important decisions, and they are algorithms that are secret and unfair. O’Neil has talked about these sorts of algorithms on her blog and in her column for Bloomberg View.
In a joint work with the philosopher Hanna Gunn, O’Neil has been developing an ethical matrix to fight against the opacity of dangerous algorithms that have real consequences in peoples’ lives.
O’Neil particularly addressed the use of creepy algorithms in recidivism models leading to race based inequality in the justice system, and in child abuse prediction models leading to disproportional risk for families living in poverty.
Algorithms predicting child abuse sometimes return false positives, meaning that a child is removed from a home where they were perfectly safe, and sometimes return false negatives, where a child is not removed from a home when they should have been. Both are bad.
The ethical matrix tries to answer: What is a comfortable rate of false positives to false negatives?
This is a difficult question to answer.
O’Neil ended with a call to action: We need to stand up against bad statistics and bad science before the trust is lost.