Recently I received a referee's report on an article. It started off as follows: 'On the bottom of page 1, the manuscript says that the method of taking action [...] is not specified; it is unclear what the author means here.' It continued for fourteen paragraphs, each filled with criticisms.
In my career, I've received hundreds of reports from referees, and their most predictable feature is criticism, sometimes relentless and merciless. Some reports start with a sentence or two saying what's nice and then proceed with the bloodletting.
Such reports can be damaging to morale. I know senior scholars who rarely or never submit articles to refereed journals because of the fierce criticism they've received. I know others who, having submitted excellent papers, were so daunted by the resulting criticism that they did not pursue the editor's advice to 'revise and resubmit,' either giving up or submitting elsewhere.
If experienced researchers can be so affected by biting criticisms, imagine what it's like for someone beginning a research career. I suspect that many talented individuals have been lost to research because of damaging reports from referees.
There is a lot of psychological research showing that most people think they are better than average in all sorts of areas, from driving to social skills to teaching, so undoubtedly most authors overrate their work. This helps explain why critical reports can be such a shock.
As well as being damaging to morale, negative reports are usually unhelpful: They do not explain well what the author needs to do to improve the article. 'The literature review is incomplete'; 'The central thesis of the paper is confusing'; 'The methodology has gaps.'
For some referees, the chance to comment on someone else's work seems to open an emotional floodgate, allowing the venting of hostility. Anonymity assists: Some friendly colleagues are capable of amazing vitriol in their reports.
Anyone with a lot of experience has stories to tell. My favourite is the referee who didn't even write a report but instead wrote impassioned comments on the manuscript, including 'bullshit.' Rejected by that journal, the paper ended up being one of my most widely read, reproduced, and translated publications.
Another factor is the scholarly culture of criticism: Students are trained to be critical of their own work, and both aspiring and established scholars build their own status by cutting others down to size. It can become second nature to look for things to criticize.
The whole idea of quality control via peer review is to weed out inadequate work. So finding shortcomings can be taken up as a challenge.
Exceptions to this pattern - the supportive and generous reports - are things to treasure. Actually, helpful reports are more common than most authors realize, because a single unpleasant report sticks in the mind much longer than several nice ones.
Many editors try to moderate the nastiness of referees, and occasionally even overrule them. On the other hand, some editors can be brutal in their comments and decisions.
The nicest rejection I've received said that lack of space was the reason: 'The responses I received on your paper were generally positive, favouring publication, but not with sufficient enthusiasm to persuade the final committee making decisions.'
After years of writing reports in the standard critical mode, I started rethinking what I was doing. Or perhaps, as I got older and had more publications, I became more generous. Anyway, I set myself the goal of writing helpful reports. I don't always succeed, but sometimes authors and editors send their appreciation.
I have two rules for writing a helpful report: Say what's good about a piece of work and tell the author how to improve it. Nothing else - no freestanding criticisms.
The first part is to describe the good aspects of the paper. There are many possibilities: a concise abstract, an interesting topic, intriguing data, appropriate methods, a well-structured argument, clear language, provocative conclusions, sensitive referencing.
Here is a sample, omitting specifics, from one of my reports:
This is an excellent analysis of theory and practice in relation to [...]. The critique of formulaic positions is well done, as is the discussion of [...]. The four responses to the question of [...] are astute and serve very well to highlight creative ways of thinking. The second half, on [...], is even more stimulating. The role of [...] is explained vividly, with informative commentary on issues such as [...]. This material shows how practical experience can provide directions for transcending sterile theoretical debates. The article is very well written, logically structured and a pleasure to read. Use of sources is appropriate.
It can take a bit of practice to say good things about a paper. The temptation to use barbed compliments - 'This is an excellent analysis of theory and practice except for the connection between them' - should be resisted.
For an author, a description of good things serves two purposes. First, it gives reassurance that something in the work is valid or worthwhile; second, it identifies positive aspects of the article that are worth retaining and building on. The risk of a largely negative review is that the author may scrap the good with the bad.
The second part of a helpful review is a precise description of what needs to be done to make the paper better. This may be on a large scale: 'A much expanded data set is needed to draw convincing conclusions'; 'References from the field of [...] should be cited and the key ideas from these references used to reframe the argument.'
Saying what needs to be done may also be specific:
Instead of the last two paragraphs of the article, it would be more helpful to have a separate conclusion section with a summary of the key points made in the article, an assessment of [...], and possibly some comments on how insights from this assessment might be used in other parts of the world.
There can also be specific comments about unclear sentences, missing references, contradictory arguments, and the like.
The big challenge in making such comments is to phrase them in terms of how to improve rather than simply stating what's wrong. Rather than saying 'This sentence is confusing,' say 'Rewrite this sentence to make it clearer.' This may seem a trivial matter, but it reflects a mode of thinking: By phrasing comments in terms of actions for improvement, we focus attention on what the author should do, not on the author's inadequacies. Focusing attention this way tends to make comments more precise - and more helpful!
That's it: the report is composed of comments on what's good and what should be done to improve the paper. Nothing else.
I normally divide my comments on improvement into two main types - those I think are essential and those I draw to the author's attention for information but not requiring action. Distinguishing essential and non-essential points is vital to help the author to decide what is top priority for revisions.
After the main comments, I often have a list of small points, from missing words to additional references for consideration. This is a good place to add some positive comments: 'Table 1 is very helpful'; 'The first paragraph in the conclusion is well expressed.'
Writing a helpful report doesn't make everything easy for the author. After all, revisions still have to be done. A request to 'redo the data analysis with a different set of criteria' is not substantively different from the criticism that 'the criteria for the data analysis are totally inappropriate.' Furthermore, a sensitive author may still feel mortified at the inadequacies implied by suggestions for improvement. The main difference is orientation: A helpful report is oriented toward building on strengths and improving weak points.
Referees are usually asked to make a recommendation, for example, 'accept without revision,' 'accept with minor revisions,' 'revise and resubmit,' or 'reject.' Everything I've said about writing a helpful report is compatible with making a recommendation.
Then I read an article by Scott Armstrong about peer review that convinced me not to make recommendations. Instead, I include the following paragraph in my reports:
Scott Armstrong (Wharton School, University of Pennsylvania) for many years has studied and reviewed research on peer review. In order to encourage innovation, he recommends that referees do not make a recommendation about acceptance or rejection, but only comment on papers and how they might be improved, leaving decisions to editors. (See, for example, J.S. Armstrong, 'Peer review for journals: evidence on quality control, fairness, and innovation,' Science and Engineering Ethics, Vol. 3, 1997, pp. 63-84.) I am persuaded by his arguments and hence make no formal recommendation about this paper.
If the editor wants to rely entirely on the referees' views, it is usually pretty easy to extract a recommendation from my comments.
In the course of my studies of suppression of dissent, I have come across stories about anonymous review being used for unsavoury motives, such as blocking a competitor from prior publication or obstructing work for ideological reasons - or, sometimes, out of envy and vindictiveness. I wanted to make sure that no one could accuse me of stabbing them in the back, so for a long time I have waived anonymity.
A few journals invite referees to make confidential comments to the editor in addition to a report for the author. I avoid making confidential comments, because this too is open to abuse.
In my reports I include this sentence: 'As is my standard practice, I do not wish to be anonymous, and would be happy to correspond directly with the author(s).' A few authors do contact me, sometimes for advice and sometimes just to say thanks for a helpful report.
Finally, I'd like to give my appreciation to all the referees who have commented on my papers - yes, even the most critical referees. There is a lot of work involved. But, surprisingly, there is very little training on how to be a good referee. Furthermore, referees seldom receive any feedback. Editors sometimes quietly drop nasty or incompetent referees, but otherwise there is little to promote change in the anonymous habits that can persist for a career.
Much more needs to be said publicly about how to write referee reports. Meanwhile, I prefer to err on the side of generosity.
Brian Martin is a professor in the Arts Faculty at the University of Wollongong, Australia. He has published more than a hundred refereed articles in the sciences and social sciences, had more than a hundred rejections, and been a referee for more than a hundred articles.
I thank Juan Miguel Campanario for helpful comments on a draft.