Peer Review

Ben Goldacre links to this piece in the New Scientist on peer review. As he points out the process isn't and in fact can never be perfect. How peer review works differs across disciplines and even within disciplines across journals. I've refereed for journals that use a double blind method and also journals where the names of authors are revealed to the referees. In sociology I would estimate that the double blind or total anonymity model is the method most commonly used. Is this the best method to use? I'm coming round to the view that we can improve things at the margin by moving to a system of complete openness. Referees should know the names of authors and authors should know the names of referees. Moreover after the review process is finished everything - the original submission, the referees reports as well as the final version of the paper -  should go into the public domain. To some eyes this will appear to be pretty radical, even lunatic, stuff. But I believe it will solve some (not all) of the problems that are chipping away at the credibility of peer review.

Firstly, as every editor knows, there is the problem of getting good referees to act and then of motivating them to do a good job. I'm told that in the US the idea of 'service' still leads the top scholars to play an active part in journal refereeing. From this side of the pond things look different. The RAE has changed the structure of incentives so that at the margin you will be better off spending an extra hour on your own work rather than writing a referee's report on somebody else's. Spending a lot of time helping your rivals - because that is what the RAE turns those you formerly called your colleagues  into - produce better work is not much of a career enhancer. 

A few years ago I was asked to referee a paper by a well known British journal. The paper was pretty hopeless, but, I felt, at least compared to much of what was published in that journal, the authors were  making an honest attempt to do science. I wrote a five page report telling them how to rewrite the paper and how to fix the errors of technique, method and logic they had made and recommended R&R. Twelve months later the paper came back to me with a different title, restructured along the lines I had suggested. It still wasn't a great paper, but at least it was now much more professionally put together. I recommended publication. It duly came out and then, to my surprise, was nominated for a prize  that is awarded annually to the best paper published in the journal. At this point the little red devil on my shoulder was whispering in my ear that I should be entitled to a share of that prize or at least a coauthorship!

If referees reports are in the public domain then that will incentivize people to do a good job and secondly the contribution of referees to the final published version will be open for all to see. This sort of openness will also help to solve what I perceived, when I was an editor, as an emerging problem - the growth of premature submission. Again the RAE is partly to blame. The publish or perish culture incentivizes people to submit articles that are in reality only seventy-five percent finished. The ends aren't tied up, the data analysis has holes in it the size of the Grand Canyon and the authors just haven't bothered to take the time to write it up properly. They know that nobody apart from the editor and the referees is going to see this version so given that almost all papers that are finally published get an R&R it makes sense to send in something that just scrapes over the R&R threshold and let the referees tell you what you have to do to achieve publication. This is not a fantasy. I've heard doctoral supervisors  give their students exactly this advice. Don't 'waste time' making the article any good. Just make it credible enough to get an R&R. I suspect in many cases they apply the same principle to their own submissions. In a cut-throat world it can quickly become a race to the bottom and intellectual craftsmanship becomes a luxury that few can afford. If my diagnosis is correct then requiring the original submission to be in the public domain will provide - as long as people have any pride in their work - a modest incentive to make the first submission serious.

My final argument for openness is that it will help to reduce some of the more unpleasant aspects of anonymous refereeing. Sad to say, some referee's reports are blatantly unfair and self-seeking. You all know what I mean. 'The article should be rejected because it failed to mention the brilliant insights contained in  a forthcoming paper by Blowtrumpet et al (currently only available as a pdf at an obscure website somewhere or other)'. You look up the paper and see that it is either irrelevant or nonsensical but the editor who is pressed for time or doesn't know any better requires that to get published you must blow Blowtrumpet's trumpet for him. There are, of course, worse things than that going on and openness won't solve them all. But requiring referees to put their names to what they write and letting everyone see what they claim seems to me to be an important part of making the refereeing process fairer. If you are going to say negative things about somebody's work you should not hide behind the veil of anonymity. I'm prepared to stand behind what I write and argue for it in the public domain. If you are not then you shouldn't be writing it in the first place.