Last week the New York Times ran the following headline: “Surprising New Evidence Shows Bias in Police Use of Force but Not in Shootings.” The article addresses a study currently underway by Roland G. Freyer Jr., a Harvard University economist. The upshot: blacks are 20% less likely than whites to be shot by police during an arrest.
Multiple studies have recently come out examining racial bias in policing. Image: Albert Barnes via Flickr.
The counterintuitive study lands amidst several other data-driven attempts to understand police use of force in a racial context, and as usual, FiveThirtyEight has done their due diligence in weighing the pros and cons of various studies and methodologies.
Fryer’s study, which culled data from a small sample of police narratives from arrests in the city of Houston, attempts to tabulate various uses of police force according to race. Statisticians across the internet have had a lot to say about the study, namely, that its findings are implausible, the statistical methods lacking, and shame on the New York Times for hyping up a purportedly paradigm shifting study that hasn’t even been peer reviewed yet — here I paraphrase the sentiment of The Statisticians.
What is seems Freyer is attempting to do is disentangle racial bias and statistical discrimination. But what’s getting lost in the data, according to Josh Miller, is the huge and variable amount of bias already present in police stops. If an officer is more inclined to perceive a black individual as threatening when they actually aren’t, Miller points out, then the actual average threat level of the blacks being arrested versus the whites might skew low. And in this case, he says, it’s no longer the least bit counterintuitive that blacks are 20% less likely to be shot. It actually makes total sense.
In one portion the study also normalizes across several parameters: arrest demographics, year of arrest, threat level of the encounter, to name a few. This is standard practice when massaging data for statistical analysis, but Uri Simonsohn points out that this normalizing actually erases all potentially salient points of the study. Particularly the perceived threat level, which the researchers refer to as “encounter characteristics.” When normalized for threat level, whites and blacks are equally likely to get shot by police. Simonsohn is quick to add that this is not robust enough to be a result on its own, but is certainly enough to cast serious doubts on the results of the study.
Somewhere along the way in life I learned that when you read a grabby headline that claims to flip the common narrative on its head, you should read the article that goes with it. And you should read it carefully, and you should be circumspect. Because as much as Malcolm Gladwell would have us believe that everything we thought was true is actually the opposite, it’s just not usually the case. True, crazy counterintuitive things happen sometimes, but flawed experimental design and sampling error happen all the time.