Refereeing a Paper

Question

I've refereed at least a dozen papers in my (short) career so far and I still find the process completely baffling. I'm wondering what is actually expected and what people tend to do...

Some things I've wondered about:

1) Should you summarize the main results and or the argument? If so, how much is a good amount. What purpose does this serve?

2) What do you do when the journal requests you to evaluate the quality of the result (or its appropriateness for the journal)? Should one always make a recommendation regarding publication?

3) What to do about papers with major grammatical or syntactical errors, but that are otherwise correct? Does it matter if the author is clearly not a native English speaker?

4) On this note, at point does one correct such mistakes? Ever? If there are fewer than a dozen? Should one be proof-reading the paper?

5) What do you do when you do not understand an argument? Does it matter if it "feels" correct. How long should one spend trying to understand an argument?

6) What to do about papers that have no errors but whose exposition is hard to follow?

Answer 1

I've been spending a fair amount of time editing a journal this year, and it's pretty amazing what different people think of as a "referee report". The first thing you should keep in mind is that the editors will be incredibly appreciative if you look at the paper in detail and send in comments in a timely manner, whatever the comments are. In my mind a good referee report begins with a very short (a few sentences) summary of the result and the argument. It includes an opinion on whether the result and proof are (i) correct (ii) readable (iii) interesting to lots or only a few people; also (iv) a recommendation on whether it is good enough to appear in the given journal, or alternatively/also comparisons to the quality of one or two other recent papers in the field, or just a statement that the referee isn't sure if it is good enough or not, and (v) a non-empty list of specific corrections/suggestions.
Re (3-4) it's perfectly fine to send in a one-line request that the paper be revised so that it is written in correct English. It's not your job to correct grammatical mistakes if there are more than a few. Re (5-6) if an argument is hard to follow, you can just ask for a revision in which the authors explain more. As an editor, it's quite easy (with computers and e-mail being what they are) to request a revision, even after the referee has only read part of the paper.

Answer 2

I think your question is so important as to deserve multiple answers, even if there is a good deal of overlap among them. (Indeed, overlap indicates that various respondents feel the same way about something, which is important.) So:

1) Should you summarize the main results and or the argument? If so, how much is a good amount? What purpose does this serve?

Absolutely yes. How much is hard to say: one thing I often do is try to put the paper down and write out a summary of the results (including precise statements) and the basic strategy of the proofs. This gives me something very longwinded. Then I try to pare it down until the point where cutting out anything else would be an obvious disservice to the author or to my explanation. It serves several purposes:

a) You need to have a high level grasp of what is being done in the paper. Writing out this summary allows you to gain this grasp (and this is especially important if the authors have done a bad job of indicating the importance of the paper in the introduction; I find that perfectly good papers are often not properly explained in this way) yourself, and

b) You demonstrate to the author and editor that you have actually read, absorbed and digested the paper. (Or you demonstrate that you haven't!) This is important both as part of the decision process and also psychologically for the author: it is really maddening to get a paper rejected by a referee who you think didn't properly read it or understand it.

c) This factual information forms the basis for the opinions you will render later on.

2) What do you do when the journal requests you to evaluate the quality of the result (or its appropriateness for the journal)?

Then you evaluate the quality of the result and its appropriateness for the journal! This is the most important part of the refereeing process, and you should do it even if you aren't explicitly asked to. (It is very unlikely that the additional opinions you provide will be unwelcome. It is much more likely that you just didn't get sufficiently explicit instructions for whatever reason.)

Should one always make a recommendation regarding publication?

Yes. It needn't always be "absolutely yes" or "absolutely no" -- i.e., you can recommend publication conditional on the author's making certain changes (possibly without looking at the paper again, I mean) or you can recommend publication more or less strongly according to certain information that the editor has and you don't (e.g. backlog), but you should definitely make a specific and unambiguous recommendation.

3) What to do about papers with major grammatical or syntactical errors, but that are otherwise correct? Does it matter if the author is clearly not a native English speaker?

This is a tough one. I think if the errors are truly grammatical and/or syntactic and they are not so serious that they create ambiguity in the meaning or interfere with your undertanding of the paper, then this should be regarded as a "copyediting" issue and be clearly placed at a lower level of importance than matters of mathematical correctness or worth.

As an author, it can be annoying to have the recommendation for publication delayed by a referee who wants to make a big fuss about copyediting issues. (Even more annoying are referees who delve into issues of formatting. I have had referees ask me why certain things are italicized and one referee who sent me an entirely new copy of my paper with the font size changed. I felt that was silly.) By coincidence I got a referee report just this evening, and -- while it did point out a matter of content, for which I am quite grateful -- there were a lot of nagging little things, e.g. "Your notation for cyclic group seems odd to me." Okay, thanks for sharing.

Ideally, there will be an actual copyeditor who steps in after the paper is accepted. Some journals do this (mostly AMS and MAA journals, if memory serves).

4) On this note, at what point does one correct such mistakes? Ever? If there are fewer than a dozen? Should one be proof-reading the paper?

That's totally up to the discretion of the referee. You certainly have the right to do it but are not obliged to. I have myself done it sometimes for authors who are both junior and non-native speakers of English, because I think that they probably will both be happy to receive it and learn from it.

5) What do you do when you do not understand an argument?

Well, that's your biggest worry when you're refereeing a paper, isn't it? I can't give a complete answer here. Certainly you should be prepared to put a fair amount of effort into understanding a crucial point in the paper, including asking friends and colleagues about background knowledge, if appropriate. If after a while you still don't get it, you should say so in the report, and in my opinion you should make clear to what extent the author is at fault for the lack of understanding and what you want to be done about it. As an author, I have gotten comments from referees like "I don't understand the proof of Theorem 13." What am I supposed to do with that information? Tell me exactly what you didn't understand, or tell me that the whole argument is hopelessly vague or flawed, or tell me that you were honestly in a little over your head at this point. But tell me something.

Does it matter if it "feels" correct?

Yes, I think it does, but I don't feel comfortable saying more.

How long should one spend trying to understand an argument?

Again, there is no one good answer to this.

6) What to do about papers that have no errors but whose exposition is hard to follow?

I would say that if you can successfully ascertain that there are no errors, the exposition was not that hard to follow! Seriously, you can make as many specific suggestions/demands for expository changes as you like, but if you have actually understood and verified the paper, then rejecting it just because it is badly written is not a decision I would feel comfortable making as a referee. (As an editor, yes.)

Answer 3

I've refereed at least a dozen papers in my (short) career so far and I still find the process completely baffling. I'm wondering what is actually expected and what people tend to do...

I used to think that I had to check a paper's correctness, but now I think that the main point of my report is to help the editors decide whether they should accept or reject the paper. It's the author's job to make sure the paper is correct. This has few practical consequences:

• I'll agree to referee a paper even if I don't think I'll be able to understand everything in it, but I'll send it back as soon as possible (within a week or two) if I feel that I can't even tell whether it's interesting or not. In this case, I'll suggest some other names.
• I do summarize the main result, so that the editors know what it's about. I try to do this in half a page. If I'm not able to do this, it's a bad sign. It has happended that I couldn't tell what the author was trying to do.
• I give a short opinion about acceptance or rejection, it's usually not hard to decide.
• After that, I include a list of the errors and typos which I found when reading the paper.

Because of the way I do this, I'm usually partial to a good introduction.

Here are a few things which have displeased me (as a referee) about the process:

• I recommended that the paper be rejected because even though it's close to my subject, I couldn't understand anything, but the editors said that they had sollicited the paper and they felt obliged to accept it.
• I made a four-page list of errors and suggested corrections, which the author ignored, and after a couple of back-and-forth, the editors asked could they please just accept the paper as it was.
• The editor was a friend of the author and felt I should not have been so harsh.

I agree with Richard Borcherds' recommandation that one should write assuming that the author will find out the referee's name. But recall that the author will see your report in a quite different light. I once started a report with something to the effect that the paper was very good and useful and bound to become a reference. A few weeks later, I was talking to the author who said the referee had written a "quite negative report".

This is the way I do things, but I have to say that receiving a six-page report where the referee has checked all my computations (including the signs!) is quite wonderful.

I think it would indeed be helpful if each journal was more precise in saying what they expect of the referee. Conversely, as a referee, I like to know the outcome (some journals received my report and that was the last I heard from them). Sending a reprint of the paper once it has appeared is a nice gesture.

My last comment is a reiteration of the fact that a timely report is important and useful. If you can't do it in two or three months, you've got to ask yourself if you are going to do it at all...

Answer 4

There are only 2 rules that really matter when refereeing:
(1) Either referee it within a couple of weeks, or reply immediately saying you do not have time.
(2) Assume that the author will see your report and will find out who wrote it.

Answer 5

1) Should you summarize the main results and or the argument?

In general, I would say: No. However, for some journals I have come to know the overwhelmed editors who in fact need this summary. So: Write it for the editors, if you write it at all. [But see my acquiescence to Jukka Suomela's compelling point in the comments.]

2) What do you do when the journal requests you to evaluate the quality of the result (or its appropriateness for the journal)? Should one always make a recommendation regarding publication?

This requires you to know the standards of the journal in question. If you do, then definitely evaluate according to those standards. For example, I review for Discrete & Computational Geometry and for Geometriae Dedicata, which both have high standards and long backlogs. I take that into account. Does this paper deserved to be published in the all-too-few pages that these prestigious journals can provide each issue, rising above many others?

3) What to do about papers with major grammatical or syntactical errors, but that are otherwise correct? Does it matter if the author is clearly not a native English speaker?

I view my job as referee to evaluate both the quality of the mathematics and the comprehensibility of the writing. I sometimes will edit just one paragraph for style and clarity (with maybe ten what I think are improvements), and suggest that this be done throughout. But it is not the job of a referee to be a style-editor, nor a 2nd-language improver.

4) On this note, at [what] point does one correct such mistakes? Ever? If there are fewer than a dozen? Should one be proof-reading the paper?

I may be abnormal in this respect, but my referee reports are not uncommonly $\ge5$ pages in LaTeX.

5) What do you do when you do not understand an argument? Does it matter if it "feels" correct. How long should one spend trying to understand an argument?

Yes, it matters! Say it "feels correct" but you have not verified it in detail. The question of "how long" depends very much on the perceived quality of the paper. I have spent upwards of 50 hrs on what I thought was a seminal paper, but have dispensed with equally long papers in an hour or two if I thought their results were pedestrian (even if correct). [I originally said 100 hrs, but upon reflection I think it only felt like 100 hrs.]

6) What to do about papers that have no errors but whose exposition is hard to follow?

You say exactly that, detailing where and why. How to proceed from here is an editorial decision, not a referee decision.

---

Finally, I will say something perhaps controversial, but not apropos RBega's question. I think anonymous refereeing is funadmentally defective in the abuses it invites, and therefore I only review nonanonymously. But I am sufficiently senior to render this without risk.

Answer 6

The referee must check if the result is correct! That is, the referee should check the proofs. This point seems to be so under-estimated, but in my opinion it is the most important part of the referee's job. The referee is the only one who will do this, except the author. There does need to be a second check of correctness after the author. There are many flawed papers out there. This has cost many other researchers time and health.

If the referee is unsure, he should ask the author for help in understanding the proofs (and the author should be helpful). It can be a change in the paper or a note for the referee.

As for whether or not the paper is "interesting": I decline to referee a paper if it is not interesting (in a broad sense) enough for me to put the effort in checking whether or not it is correct. Thus if all referees were correctness minded, then "interesting" papers would get their advantage.

Answer 7

Be timely (if you can't do it in <6 months then don't agree to referee). Be accurate. Don't take the author's reputation into account when rejecting or accepting an article, and insist on good exposition. Remember also that your report is a report to the editors, not the author.

Answer 8

I feel that I more or less agree with everything that has been said so far (so in a sense I don't know why write this), but personally, I find it really annoying when the type-setting and grammar are really really poor in a paper that I've accepted refereeing (maybe I shouldn't have accepted then you might say).

If the authors haven't spent the effort making the paper readable and pleasing to the eye it all looks so sloppy and that doesn't inspire confidence in the paper. I know this is wrong of me, but that's the way I feel.

In practice though (this has happened more than once) I try my very best to give the paper a fair and unbiased review and I also indicate in my report most of the typos and grammatical errors I find, even if this is somewhat pedantic. I feel that my job as a referee is also to make the paper readable to others, in addition to error-spotting.

Answer 9

I am not sure if this point has already be raised: If your report criticize something or votes for a revision, then give hint what the authors can do. Comments of a referee which are particularly useless are e.g.: "The presentation is messy" or "The references are not appropriate".

Answer 10

Re question 5 ("How long should one spend trying to understand an argument?"): an interesting case was Hales's proof of the Kepler Conjecture, in which the 12-member team of Annals referees spent five years before resigning themselves to being unable to completely certify the validity of the proof.

Answer 11

I've had long discussions about whether the referee report should be author-dependent. If mathematician X and mathematician Y submit the same paper, should the report be the same?