Every day the average American consumer comes into contact with hundreds — if not thousands — of algorithmically powered experiences.
The ads one sees on Facebook or Google were served courtesy of an algorithm that mashes up data points about what you have searched for in the past and where you have shopped. The blazer, blouse and shoes recommended for you by your favorite eTailer was also decided by an algorithm that mashed up data points about what you bought in the past and their inventory.
Some algorithmic experiences, however, are much harder to miss. Anyone who has ever applied for a car loan, a mortgage, a credit card, a job online or to college knows that algorithms — and the decisions that they inform — can make very big differences to extremely important life decisions.
And that, author and Harvard Ph.D. mathematician Cathy O’Neil told a rapt audience at IP 2017 last week, can lead to a very dangerous situation when algorithms go wrong, and often do. Sometimes, they go so completely pear-shaped that she says they become WMDs: Weapons of Math Destruction.
What is a Weapon of Math Destruction?
According to O’Neil, who literally wrote the book on the subject, an algorithm goes full WMD when it has widespread (used often), mysterious (poorly understood outside of technician circles) and destructive (do recordable damage to people’s lives) results.
The problem, according to O’Neil, is that to build a good algorithm one needs to do math, which leads people to jump to the erroneous conclusion that algorithms themselves are as neutral as the math that is their origin. But that, O’Neil said, simply isn’t true. An algorithm is a model coded with math, and models are built using data with outcomes that are defined by conditions for success.
And, O’Neil asserts, those conditions for success are often opinion-based and in need of auditing, just like any other opinion.
But algorithms aren’t billed as a sophisticated rendering of opinions. They are billed as natively objective and fair, which, O’Neil noted, her time on Wall Street in the early 2000s quickly proved they weren’t.
“I saw algorithms be totally manipulated when it came to triple A-rated securities. They were just mathematical lies,” she told the crowd at IP 2017 at Harvard.
And these types of “lies” abound, O’Neil said, though they appear for all types of reasons. In the case of Wall Street, the goal wasn’t to build a good model that successfully predicted risk; it was to build a model that would allow profits to be maximized — and those success conditions dictated what they built their code around.
Sometimes, she explained, the intention isn’t to deceive. The algorithm itself is just badly designed; it’s just not designed well enough to accurately predict anything at all. O’Neil referred to the teacher-ranking algorithms used in firing and tenure decisions in about half of the states today as part of educational reform. The problem with that algorithm is that it isn’t even internally consistent — and it will rank the same teacher good and bad in the same year if they happen to be teaching two sections of something to two different grades.
The good news about those kinds of bad algorithms, O’Neil noted, is that they don’t last long in private industries. But they can be very dangerous when people become so strongly dedicated to them for political reasons that they stop caring that they obviously don’t work at all.
But the most dangerous type of WMD is the one that’s invisible, because, unlike the other two, there is nothing obviously wrong with it. It is an algorithm that’s built out of a perfectly reasonably seeming data set with a perfectly reasonably seeming set of success conditions or outcomes.
But those reasonable conditions can mask some deep biases in systems and individuals. And those biases can produce results that are not only very harmful, but also hard to argue with, because they seem to come from a neutral, well-programmed algorithm.
When Algorithms that Want to Be Good Go Terribly Wrong
O’Neil stated that we as a society have very little information on crime itself, because people don’t generally admit to committing crimes. Instead, what we have is information on arrests, which stand in as a proxy for crime.
Predictive policing, O’Neil notes, focuses on sending police to high-crime neighborhoods, which translates as neighborhoods where there are historically lots of arrests (typically minority neighborhoods). As a result, she says, crime data is overly skewed toward minority communities and under-reported in non-minority communities.
“Crime data could look very different right?” O’Neil asked the audience. “Like if cops had marched in and arrested everyone on Wall Street for the financial crisis — that data would look very different. When it comes to crime, there’s just a lot of missing ‘white’ data.”
The situation in financial services, on the other hand, is the mirror image: “There’s a lot of missing minority data,” O’Neil continued.
And that missing data has consequences for the financial lives of those communities that exist in the data black hole. Among white consumers, six to seven percent of mortgage applications are denied; for Hispanic consumers, that figure jumps up to 12 percent; and for African Americans, it’s 17 percent. Eighty percent of white consumers have access to credit cards, as opposed to 70 percent of Latino consumers and 53 percent of African American consumers. Minority communities also tend to pay higher interest rates than their white counterparts. According to the CFPB, minority borrowers on average pay two percent to three percent more interest on auto loans than their white counterparts.
And, O’Neil noted, much of that difference comes via a data deficit. Credit rating agencies tend to have more relevant data on white Americans than minority Americans, which means white Americans tend to de facto have higher credit scores.
The algorithm panel that joined O’Neil on stage to discuss how to disarm those WMDs agreed that problems in finance with modeling get even more complicated quickly.
Nitya Sharma, co-founder & CEO, Simpl; Carey Ransom, chief product officer at Experian Consumer Services; Jai Holtz VP at Vyze, former president of Sears Financial Services; and Kathleen Pierce-Gilmore VP & GM Credit Americas at PayPal all concurred that lack of inclusiveness is a problem for financial services.
It’s business problem, because, as most of those on the panel noted, the goal is to lend out more money and find more people who will pay it back. Often pressure to build size fits all products, because they are using algorithms that are picking too narrowly — meaning they aren’t doing as much business as they should because they are offering too narrow a range of services to a very narrow set of potential customers.
And, the panel noted, it is also an ethical problem, because it is not sufficient for financial services to look at communities where there isn’t enough data and offering “no credit for you” as a response. The goal is to look for deeper data sets, and data sets that allow for better predictions for a wider swath of consumers.
Those data sets aren’t easy, the panel noted. There is a risk at looking at the wrong kinds of information and using data to further buttress biases (about consumer wealthy, or race, or preferences or gender) instead of using more data to see customers more clearly.
There is also the reality of what the panel called “legislatively enshrined bias” where lenders have to factor in information about borrowers, even if they know it isn’t really well correlated to underwriting outcomes. One panel member noted that a lot of lending regulation is pushed toward lending to consumers who already have money, which means that there are real limits to what can be offered to lower income consumers.
But, the panel noted, this is also where innovation can make a difference. Up-and-coming players can work directly to address niches in the marketplaces because their focus in a specialized data set — and their success conditions are greater inclusion for the underserved.
No one wants to be biased. O’Neil noted that very few mathematical lies are actually told by people who expressly want to deceive. But, she notes, it is easy to believe something false and then program an algorithm to only search for data that proves our bad conjecture.
More data and better data is part of the solution. Zeroing in on the outcome first and then devising the model to drive that outcome is critical.
Free of biases or data that may inadvertently reflect them.