Theoretical computer scientists have been talking about double blind peer review, and it’s an interesting discussion. The current incarnation of this discussion started when Rasmus Pagh and Suresh Venkatasubramanian used a double blind refereeing process for submissions to the ALENEX18 conference they co-chaired. Venkatasubramanian posted about their motivations and how they pulled it off in two posts on his blog, The Geomblog (post 1, post 2, post 3).
Artist’s rendition of blinded peer review. (Justice, from the Cardinal Virtues by Nicolaes de Bruyn) Credit: Public domain, via the Metropolitan Museum of Art
Why double-blind? First, it’s the standard for computer science conferences outside of the theory subdiscipline. More importantly, many people worry that single-blind peer review, where the reviewer knows the identity of the author, leads to some objectionable outcomes based on implicit and explicit biases. More famous authors or authors from more prominent institutions may have their work reviewed more favorably, and more broadly, the bias in favor of these authors combined with other biases reviewers have can continue systemic bias against women and other groups that are underrepresented in the field.
Obviously, a major change in the paper submission system is not without controversy. The discussion has continued in posts by Boaz Barak, Michael Mitzenmacher, Omer Reingold, and Lance Fortnow. In general, the conversation I have seen has been civil and thoughtful. In one post, Venkatasubramanian writes,
First up, I think it’s gratifying to see that the the basic premise: ‘single blind review has the potential for bias, especially with respect to institutional status, gender and other signifiers of in/out groups’ is granted at this point. There was a time in the not-so-distant past that I wouldn’t be able to even establish this baseline in conversations that I’d have.
“The argument therefore has moved to one of tradeoffs: does the installation of DB review introduce other kinds of harm while mitigating harms due to bias?
A few math journals—mostly in math education and undergraduate research, as far as I can tell—do use double-blind peer review. But it is not standard. One of the biggest barriers to double blind reviewing in computer science, physics, or math is the fact that so many preprints are posted on arxiv or authors’ websites before they are submitted, making it that much more difficult for a reviewer to avoid knowing who wrote the paper. (Venkatasubramanian writes about how they dealt with that problem in his posts; one point he makes is that double-blinding the process won’t necessarily prevent reviewers from being able to determine authors eventually, but it could prevent some knee-jerk reactions. He also points to a post by Regina Barzilay that delves into the issue in more depth) In some fairly narrow subdisciplines, there are few enough researchers that even without seeing the paper online, others in the field will be able to tell who wrote it anyway.
While societies and individual humans in them have biases, there will be no way to completely eliminate these biases when people (or algorithms) make decisions about paper and conference submissions. It is important for academics to look at the advantages and disadvantages of different strategies to mitigate the effects of bias. I am looking forward to seeing how this conversation evolves.
I think it would be a great idea, but it is not very realistic. Between subject matter, presentation style, and recurrent English language errors I bet my papers can be easily identified by anyone who knows me mathematically – i.e., anyone who would be considered as a referee. And I’m not special in this, we’re a small community and we usually also know who’s working on what. Still, it could work for younger people.
If we want to try, it should be with the support of arxiv, who should allow posting of preprints with no authors nor acknowledgements until the paper has been accepted.
All of the major MAA journals — American Mathematical Monthly, Mathematics Magazine, College Math Journal — use double-blind review.
As there may be value in comparing practices across fields, here is my experience from moving from physics (where single-blind was the norm) to philosophy (where double-blind is the norm, with sprinkles of triple-blind).
At first, it was very odd to be a referee of an anonymized paper: you find yourself guessing who wrote it. But at the time, there were influential calls for restraint on the side of the reviewer that helped to change my mind about this. Once you acknowledge that you too are susceptible to all kinds of biases, it makes sense to refrain from Googling the title, etc.
I got used to it very fast and now I find it super awkward when I agree to review something and then it turns out to be the odd journal or conference that only has single-blind refereeing.
Meanwhile, I also started grading exams in an anonymized way, where feasible. (That is: except for papers in small groups, where it gets in the way of giving students individual feedback prior to the final deadline.)
There are some philosophy journals that offer a triple-blind anonymous procedure. One of them is Ergo: you can read there peer review policy here https://www.ergophiljournal.org/review.html. They recommend to authors to remove the title of their submission in case it has appeared in a conference program.
In short: while it isn’t watertight, I do believe double-blind to be an improvement, and triple-blind a further improvement (to avoid desk-rejects by editors due to prestige bias etc.).
I have been victimized by double-blind peer review before, where the referees figured out that the paper was mine and used the prior paper to complain about the current one. While that can (and does) happen in the single-blind case as well, the fact that the editor used it against me — even amidst pointing that my paper was treated in a different way from other papers submitted to the journal — suggests that another possible risk of double-blind peer review is overreliance on its fixing problems.
I have also been a referee in double-blind peer review in which the authors figured out how I was, but I had no idea who they were (unless after the paper was published in another journal).
One essential problem* that occurred in the above two situations is the same in both cases: My scientific writing style is just so different from everybody else’s that it is systematically much easier to de-anonymize me than it is to de-anonymize other people, and I am really uncomfortable with the idea of some people being much easier to de-anonymize than others. (Imagine if it’s a de-anomyized person who has written controversial public posts about social issues, as some people who write in more unusual styles are easier to de-anomymize.) So there are times that this can unlevel certain parts of the playing field, and the balance with other playing fields that this choice makes more level is indeed one to consider very carefully.
* There are other issues, but I figured the one that is most salient to bring up is the one where it’s a systematic one that works against me personally. Obviously, I am but one (very unusual, apparently) tree in the vast forest. In the example above where I was an author, I think there is a clear solution if editors are conscientious and careful: the review that actively de-anomymized me should not have been used to judge my paper.