Revising and resubmitting my opinions on refereeing

by Orr Shalit

With time, with age, having done already quite a few paper-refereeing jobs, I have come to change some of my opinions on refereeing.

Anonymous refereeing. I used to think that anonymous refereeing was not important. Why can’t I (as referee) just write back to the authors and discuss the weak points of the paper with them. Wouldn’t that be much better and faster? Besides, if I have a certain opinion about a paper, I should be willing to back it with my name, publicly.

Yes, I was innocent and was not yet aware of the endless ways in which some people will try to get back at you if your report includes anything but praise and/or typo corrections. But besides the usual reasons for or against anonymous refereeing, here is something I overlooked.

The really nice thing about anonymous refereeing is this: not only does it free the referee to say bad things, it also frees the referee to say good things. There was a paper I was reviewing for a good journal, and I really wanted the paper to get accepted. I thought it was very good, and that it should be accepted to this good journal. I wanted to be very clear that this paper should be accepted (sometimes, a lukewarm report is not enough to get a paper accepted), and being anonymous made it easier for me to use superlatives that I rarely feel comfortable using in front of someones face. The fact that I was anonymous, and the fact that the editor knows that I am anonymous, also makes it easier for the editors to take my praise seriously.

Therefore, the identity of referees should be kept secret, so we can all be kind to each other. (And please don’t ever ask me if it was me who refereed your paper).

Is the paper interesting? When I first heard that papers get rejected because they are “not interesting”, I was a little surprised. “Interesting” is not an objective criterion. It might be interesting to one person, and not interesting to another. Certainly the author thinks it is interesting!

Certainly? Well, I have seen some papers, unfortunately, about which I cannot say that I am certain that even the author thought that they are interesting. I have seen some papers that were written only because they could be written. Nobody ever wrote that particular proof to that particular proposition, with this set of assumptions, so this is a “new contribution”. But sometimes, a paper contains nothing which has appeared before, but does not really contain anything new. If there is nothing new, then it is boring, not interesting.

It is very hard to say what makes a good paper, and what makes a bad one. What makes good scientific research? I believe that judging the value of scientific research is not a scientific activity in itself. Deciding whether a mathematical paper is good is a job for mathematicians, but it is not a mathematical problem.

So when I evaluate a paper, I check if it is correct and new, of course, but I also cannot help but thinking whether or not it is interesting. What does interesting mean? It means interesting to me, of course! But that’s OK, because if the editor asked for my opinion, then it is my opinion that I am going to give.

Do I work for the journal. The editors of Journal A say that they want to publish only the best research articles. What does that mean? How can I compare? Let me tell you if the paper is new, correct, and interesting. What do I care that Journal A wants to remain prestigious? In fact, I never published in Journal A, and as far as I care its reputation can go to hell.

And really, to be honest, there are many factors that may affect my decision to recommend acceptance of  a paper to Journal A: 1) The authors are young researchers and this could help them in their career. 2) The paper is in my field, and I want to use the reputation of Journal A to increase the prestige of my field. 3) etc., etc., one can think of all kinds of impure reasons to be consciously biased for accepting a paper. In any case, if the paper is in my opinion a good, solid contribution, then why is it my business that the editor wants only to publish spectacular papers?

I now look at it differently. It is an honour to be approached by Journal A and be asked for their opinion. The editor is asking my professional opinion, based on my reputation. I should keep in mind that my answer, among other things, affects my reputation. I have to behave like a professional, and answer the question asked. Of course, I still don’t work for the journal, and I am free to be very enthusiastic about papers that are important in my opinion.

More on the pecking order. I have heard more than once of the following scenario: an editor of Journal B tells a referee that his journal (Journal B) is now only accepting papers that would be good enough for Journal A.

Excuse me!? If the authors thought their paper was good enough for Journal A, then they would submit it to Journal A, and not to B! And anyway, I don’t work for the journal! Clearly the journal has its goals, it wants to increase its prestige (or whatever), but I also have my own priorities, and in any case I don’t care about the prestige of Journal B. I’m already being very nice that I am willing to referee this paper for free, so don’t ask me to work for your prestige (if it is good enough for me to referee, then it’s good enough for you to publish).

Actually, the idea that Journal B aims to be at the “quality” of Journal A (whatever that means) is not so ridiculous. Journal A rejects most of the papers submitted to it, in fact it rejects some excellent papers. Where are all these papers supposed to go? So I don’t mind answering the question asked. (What I once wanted to write, but did not, is this: “Yes, I would recommend it for Journal A, and in fact this paper is too good for you, Journal B! I recommend rejecting the paper on the grounds that it is too good for this journal…”)

Submitting my review in a timely manner. I have not changed my mind about that. I always give an estimate of when I will submit my report, and I always submit on (or before) time. This means that I have to say “no” to a large fraction of referee requests (I try to referee at least about as many papers as I publish every year), otherwise I would not be able to do it in a timely manner. Naturally, I try to accept for review the papers that are more interesting.