On Blind Reviewing

This is something about which I have wanted to write for a long time. Since, like many other things about which I want to write, it is quite an important matter, I didn’t want to write in a hurry. Which meant that I had to wait for a time when I could write at enough leisure to be able to write at enough length with enough time for making it rigorous enough. Now, since it is very difficult (for me at least) to get enough of all these, this effectively meant that writing about this topic was postponed indefinitely.

But I don’t want this to be postponed indefinitely. I want to write about this now. So, I would just write and try to be as rigorous as it is possible to be in a blog post written in one or two short sittings. This applies to many other posts, whether written already or to be written in future. You can take it as an apology or you can take it as a disclaimer.

What is the problem? Well, the problem, or rather the question, is whether what is called ‘blind reviewing’ is a good thing or not. And, of course, this is in the context of peer reviewing of scientific (or claimed to be scientific) research papers or articles for the purpose of selection for inclusion in the proceedings of a conference or workshop or for inclusion in a journal.

Excuse the legal sounding language.

First of all, let me list all the reasons in favour (‘favor’ for the dominant party) of the so-called ‘blind reviewing’ process, so that no one can jump and dismiss the whole affair as trivialization by saying you don’t know what you are talking about:

  1. Human beings can be biased. So, if a reviewer knows that a research paper is written by a person she doesn’t like or has strong disagreement with, she can get biased against the paper and will not be able to review the paper fairly.
  2. Apart from the above kind of biases, there can be the bias in terms of the weights associated with the names of the authors, their institutions, their countries, their group, even their academic background. Most of the people who have been working in NLP/CL[1] for some time know about the linguistics vs. statistics or machine learning bias. This kind of bias increases the chance of your paper being rejected or accepted depending on whether you seem to be in favour (or favor) of a linguistics heavy approach to NLP/CL or of a statistics (or machine learning) heavy approach. There are variants of this bias in other fields too. For the closest example, we can consider Linguistics. Where your paper is perceived to be situated along the Chomskyan or Empiricist or Cognitive or Computational axes with respect to the chosen position of the reviewer can have a large impact on the decision about your paper, irrespective of what else your paper says. And the chances of such a perception can be increased if the identities are known.
  3. Human beings can be unduly confrontational and they can also be unduly wary of confrontation. So, if the identity of the reviewer is not withheld, the author(s) may be offended by the reviewer and they may also become confrontational and carry on this confrontation with the reviewer, thus making the process of reviewing difficult and something which a lot of people would like to run away from. Also, the reviewer may avoid making adverse comments, especially if the reviewer doesn’t want to offend the author(s).
  4. If the author(s) don’t know who the reviewer is and vice versa, the whole reviewing process may be more fair for the above specified reasons and because of the general association between anonymity and fairness. If you don’t know who is criticizing and the person criticizing also doesn’t know who is being criticized, then you can expect more fairness.
  5. If the Program Committee (PC) chair(s) also don’t know who the authors are and who the reviewers are, then they can assign equal weight to all the reviews for making the final decision about a paper.
  6. If the author(s) don’t know who the reviewer is, then they won’t have any reason to attribute bias or prejudice to the comments made or ratings given by the reviewer.
  7. Peer reviewing of research papers, like the administration of justice, should not just be fair, but seen to be fair. And this can only happen with blind reviewing.
  8. Blind reviewing, through the use of the device of anonymity, gives a true meaning to the idea of ‘peer reviewing’, because if the identities are not known, all the people involved can be treated as peers, even if some of them are senior most pioneering researchers or Directors of first class institutions in first world countries, while some others are graduate students in second class institutions in third world countries.
  9. If the identities are not known, both the reviewer and the author can focus on the content of the paper and the review, respectively.
  10. Finally, the very practical reason that blind reviewing provides a reasonably fair mechanism to ensure the selection of the best research papers such that everyone can be more or less satisfied with the outcome and no one will have valid reasons to complain.

I think the above list makes as strong a case for blind reviewing as can be made. I mean in a blog post, not in a book.

Now, in the next post (that means in some future post) I will discuss what is or can be wrong with blind reviewing and will try to draw some conclusions. You must have guessed that the reason I am writing all this is that I am not sure whether blind reviewing is the best thing possible. But by writing all this, I am also trying to get things straight in my own mind.

[1]: With apologies to Martin Kay and others, I am using NLP and CL as interchangeable terms because I think my arguments in this matter are not affected by the distinction between the two, a distinction which may be important in many but not all contexts (i.e., in my opinion).

Too Good to be True

So there were a lot of things to be done in a short time and, apart from one or two minor hiccups, everything went quite well: The paper presentation, the demo, submission to the ACL main conference, and the two workshops. In fact, one of the workshops (NLP for Less Privileged Languages) was an unexpected success, both in absolute and relative terms. The funny thing is that while this workshop was the one my heart was set on, I had to do much much more work for the other workshop (NER for South and South East Asian Languages). Believe me, the ‘much much more’ part is more than justified. Conducting a workshop (when you have to do most of the work) is not very easy. Conducting a workshop that includes a shared task is tough. And a shared task which requires new training and testing data is tougher. And it’s even more tough when you don’t have any funding etc. You get the point.

So, everything went quite well.

Not quite.

After the workshop work was finished I just happened to look at the final version of the paper which was submitted to ACL. And I found to my shock that a figure which I had added and on which I had spent around two hours on the night before the workshop (my heart was set on) was missing. The caption was there and so were the references to the figure, but the figure itself was gone.

What happened was that while I was in the middle of the workshop, one of the co-authors noticed a minor typo. But he is now doing a job and is not here. He was unable to compile the Latex files on his system. So he asked another co-author to correct the mistake. The other co-author did as he was told, but he didn’t check before submitting the revised version. Apparently some library was missing or incompatible on his system and so the figure simply disappeared from the paper. Now, I had worked all night before the workshop (my heart was set on) to complete that paper and I thought I had done a good enough job. Moreover, that paper has already taken two rejections, in spite of the fact that it has taken a lot of work from more than one person. Even as an unbiased reader (to the extent it is possible for me to be), I think that paper was worth selection in a major (‘first class’) conference.

So many things to do in so short a time and everything going almost perfectly?

It was too good to be true. Something had to go wrong. Murphy’s law.

That jinxed paper seems set for another rejection. Leading to more dejection.

Just because a library was missing (or incompatible) on a system and the version submitted had one figure missing.

I don’t know whether I should say this, but had this happened to some other author and I was the organizer, I would allow the author to submit a revised version with the figure included, provided the authors had not added anything more after the deadline.

Does that sound like a crazy idea?

Hectic Days are Here

There was a lull before the storm (in a tea cup?). The easy days are over and the hectic days are here. Today I have to submit a paper for the ACL conference. Yesterday I had to present a demo at IJCNLP conference. The day before that we had a paper presentation at the same conference. And the day before that I had to attend two tutorials, again at the same conference.

Tomorrow there is an IJCNLP research workshop of which I am the organizer. The day after tomorrow there is another workshop of which I am the co-organizer. And I have to make introductory presentations at both the workshops.

After that starts the ‘real’ work which I am supposed to do. Some professional, some formally academic and some informally academic.

These must be among the most hectic days of my life. Certainly the most hectic involving public presentations.

I hope I survive.

%d bloggers like this: