This is something about which I have wanted to write for a long time. Since, like many other things about which I want to write, it is quite an important matter, I didn’t want to write in a hurry. Which meant that I had to wait for a time when I could write at enough leisure to be able to write at enough length with enough time for making it rigorous enough. Now, since it is very difficult (for me at least) to get enough of all these, this effectively meant that writing about this topic was postponed indefinitely.
But I don’t want this to be postponed indefinitely. I want to write about this now. So, I would just write and try to be as rigorous as it is possible to be in a blog post written in one or two short sittings. This applies to many other posts, whether written already or to be written in future. You can take it as an apology or you can take it as a disclaimer.
What is the problem? Well, the problem, or rather the question, is whether what is called ‘blind reviewing’ is a good thing or not. And, of course, this is in the context of peer reviewing of scientific (or claimed to be scientific) research papers or articles for the purpose of selection for inclusion in the proceedings of a conference or workshop or for inclusion in a journal.
Excuse the legal sounding language.
First of all, let me list all the reasons in favour (‘favor’ for the dominant party) of the so-called ‘blind reviewing’ process, so that no one can jump and dismiss the whole affair as trivialization by saying you don’t know what you are talking about:
- Human beings can be biased. So, if a reviewer knows that a research paper is written by a person she doesn’t like or has strong disagreement with, she can get biased against the paper and will not be able to review the paper fairly.
- Apart from the above kind of biases, there can be the bias in terms of the weights associated with the names of the authors, their institutions, their countries, their group, even their academic background. Most of the people who have been working in NLP/CL for some time know about the linguistics vs. statistics or machine learning bias. This kind of bias increases the chance of your paper being rejected or accepted depending on whether you seem to be in favour (or favor) of a linguistics heavy approach to NLP/CL or of a statistics (or machine learning) heavy approach. There are variants of this bias in other fields too. For the closest example, we can consider Linguistics. Where your paper is perceived to be situated along the Chomskyan or Empiricist or Cognitive or Computational axes with respect to the chosen position of the reviewer can have a large impact on the decision about your paper, irrespective of what else your paper says. And the chances of such a perception can be increased if the identities are known.
- Human beings can be unduly confrontational and they can also be unduly wary of confrontation. So, if the identity of the reviewer is not withheld, the author(s) may be offended by the reviewer and they may also become confrontational and carry on this confrontation with the reviewer, thus making the process of reviewing difficult and something which a lot of people would like to run away from. Also, the reviewer may avoid making adverse comments, especially if the reviewer doesn’t want to offend the author(s).
- If the author(s) don’t know who the reviewer is and vice versa, the whole reviewing process may be more fair for the above specified reasons and because of the general association between anonymity and fairness. If you don’t know who is criticizing and the person criticizing also doesn’t know who is being criticized, then you can expect more fairness.
- If the Program Committee (PC) chair(s) also don’t know who the authors are and who the reviewers are, then they can assign equal weight to all the reviews for making the final decision about a paper.
- If the author(s) don’t know who the reviewer is, then they won’t have any reason to attribute bias or prejudice to the comments made or ratings given by the reviewer.
- Peer reviewing of research papers, like the administration of justice, should not just be fair, but seen to be fair. And this can only happen with blind reviewing.
- Blind reviewing, through the use of the device of anonymity, gives a true meaning to the idea of ‘peer reviewing’, because if the identities are not known, all the people involved can be treated as peers, even if some of them are senior most pioneering researchers or Directors of first class institutions in first world countries, while some others are graduate students in second class institutions in third world countries.
- If the identities are not known, both the reviewer and the author can focus on the content of the paper and the review, respectively.
- Finally, the very practical reason that blind reviewing provides a reasonably fair mechanism to ensure the selection of the best research papers such that everyone can be more or less satisfied with the outcome and no one will have valid reasons to complain.
I think the above list makes as strong a case for blind reviewing as can be made. I mean in a blog post, not in a book.
Now, in the next post (that means in some future post) I will discuss what is or can be wrong with blind reviewing and will try to draw some conclusions. You must have guessed that the reason I am writing all this is that I am not sure whether blind reviewing is the best thing possible. But by writing all this, I am also trying to get things straight in my own mind.
: With apologies to Martin Kay and others, I am using NLP and CL as interchangeable terms because I think my arguments in this matter are not affected by the distinction between the two, a distinction which may be important in many but not all contexts (i.e., in my opinion).