In the last post I tried to put across some points against blind reviewing. The intention was to show the limitations of blind reviewing. As I said in that post, I don’t have the correct solution. But I do know now that blind reviewing is not as good an idea as it seems on the surface.
I will have more to say about this, but in this post I will digress from the topic a bit and consider a hypothetical situation.
At present, when the author(s) of a paper get the reviews, they (in most cases) have no venue to reply to the comments made in the reviews. Now, suppose we had completely open reviewing. Not double blind. Not single blind. Zero blind.
Zero blind means that the reviewers know who the author(s) are and the author(s) also know who the reviewers are.
Hard to imagine? But we already have such a reviewing process in real world. In fact, this is the most used process. This is how books are reviewed in the media. Or movies. Or anything artistic or scientific.
Yes, I know, the situation with research conferences and journals is a bit different and there are many practical difficulties. But let’s ignore them for the time being and just assume that a zero blind process is in place and a paper is sent to a conference.
Now, the reviewers do their job and based on their comments and scores (if any) it is decided to reject the paper. This is again not a universal thing, because there is also something called ‘killing a paper’ where a single reviewer can virtually ensure that the paper is, well, killed. Regardless of what other reviewers might have said. This actually happens.
But we will ignore this too for our hypothetical case and just assume that the paper was rejected after considering all the reviews and discussion among the PC members.
Now, when the authors get the reviews, they believe that the reviewers have made mistakes in understanding the paper. Also, that the reviewers have made statements which are not justified. The authors, because they know who the reviewers are, write a detailed reply and counter all the comments and statements made by the reviewers which they think are not justified.
The reviewers are also interested in the problem that the paper is about and they probably are also working on the same problem. Which is why the paper was assigned to them in the first place.
Once again, this is an assumption for the sake of idealization. In reality, the reviewers often are not working on the same problem and may not even be interested in it.
What is going to happen now? The reviewers also reply to the author(s)’ comments. This could start a confrontation which goes on for a long time and arguments are traded back and forth.
Note that the decision about the paper has already been taken because the only reform that has happened in this hypothetical situation is that we have zero blind reviewing instead of single or double blind. So the confrontation was started by the reviewing process, but it is no more a part the process now.
‘Conventional wisdom’ says that confrontations (with words and verbosity) among researchers (or, in general, members of a group) are bad. This is the same wisdom which says that confrontations (with weapons and violence) may be good among nations or communities because there are things called national interests or communal interests.
So how about collaboration? Can this exchange of arguments lead to collaboration among the author(s) and the reviewers?
Apart from practical constraints, is there any reason why this can’t happen? I mean, even if collaboration is ruled out in many cases due to practical constraints, it may still happen is some cases. Wouldn’t that be a good thing for research?
Of course, I am talking about Computational Linguistics and Natural Language Processing, where most of the action takes place on computers. Which are likely to be connected to the Net. So, collaboration, i.e., long distance collaboration, is not a Utopian dream.
And I also know it’s not just that collaboration may not happen, there may be an ugly confrontation. This might lead to worsening of social relations among the researchers. It won’t be a good situation.
But wait a minute!
Aren’t researchers supposed to be mature people who, for the most part, think rationally? Aren’t they supposed to be objective, or at least rigorously and honestly subjective? Aren’t they supposed to be good enough to take the responsibility of deciding which research paper should be accepted and which should be killed. Or not allowed to be born, to put it more politely? In fact, aren’t they assumed to have a lot of qualities which we don’t so easily assume in ordinary mortals? Such as the fact that when they recommend the rejection of a paper and make sure it’s never published, they won’t take some ideas from those rejected papers and use them for their own work as their own contribution? Unconsciously, if not consciously. To suggest otherwise (i.e., that they can be unknowingly plagiarist) would actually be considered a blasphemy. And if someone gets caught, it would be considered a great scandal. There is some social psychology involved here which I would rather not talk about right now.
If the researchers who review papers are already assumed to have all these wonderful qualities, can we also assume that in most cases they won’t get into an ugly confrontation? That they would, if possible, convert a possible confrontation into a collaboration?
It all depends on what our preferred model of relations among researchers is. If there is a four colour (color) spectrum, which one is your preferred color:
- Between Newton and Leibniz
- Between Einstein and Bose
- Between Bertrand Russell and Wittgenstein
- Between Hardy and Ramanujan
In case there is anyone reading this, and assuming that they don’t think this is all pure BS…
What do you say? Which model do you think can work? Which is your preferred model? Which model would you recommend for application in the real world?
In today’s world? But more importantly, in tomorrow’s world?