Sunday, March 29, 2015

The Peer-Review Process: Problems and Pitfalls, but it's Worth Saving!


The peer-review process for research articles is a well-established method of screening new results prior to being released to a larger audience. The results of peer-review are supposed to mean that if you read information in a peer-reviewed journal, you can be assured that what is being said makes sense and the information can be used as a sound basis for making decisions.

I should note at this point that I have reviewed articles for many international journals and am an Associate Editor for the American Journal of Enology and Viticulture. In that capacity I screen manuscripts for suitability (e.g. is it within the scope of the Journal, are there glaring errors in science etc.) before sending it out to two people with relevant experience for critical review. I take those reviewer comments, see if they are fair, and then pass them back to the authors for them to consider in revising the document. That is then submitted again and I would look to see if they have addressed all the relevant reviewer comments. If so, the paper is accepted for publication. If not, it may go back for further review, or rejected.

It's not a perfect system (it can take a long time to go through this process, you have potential biases from reviewers, it tends to be a conservative process etc.), but if you look after it well it does a good job.

All researchers that submit manuscripts for peer-review should be giving back to that process by accepting manuscripts to review for those journals -although there are a lot of questions about that process, too, as the people who do the reviewing work provide the engine that runs many for-profit publishers...

However, that aside, the model of peer-review is under fairly extreme pressure. The basis of the process is that people who have expertise in an area of research will do a critical review of papers submitted for publication. The key words there are "critical review."

In my experience, it takes some time to do this. Even screening an article before sending it out to reviewers can take more than an hour, and for me to do a proper review (which includes reading it carefully, researching problematic statements, and then typing up comments) takes about four hours or more.

Researchers have less and less time available to them for a variety of reasons - with constant reductions in budgets, we have more administrative duties. With non-replacement of staff, we're taking on more teaching. There is increased pressure to find external research dollars. There is increased pressure to publish, leading to taking on more research students. In short, we have less and less time to devote to something that produces no tangible benefit to our employers. Four hours spent doing a quality review of a manuscript is four hours spent not doing all the things that a employers wants.

So as an Associate Editor, I'm finding some reviews quite superficial. Increasingly, I also find it difficult to get someone to agree to doing a review in the first place.

As a reviewer, I still try to spend all the time that's necessary to do quality work, but this does come at a cost, personally and professionally.

This latest news item, "Major publisher retracts 43 scientific papers amid wider fake peer-review scandal," is not an isolated example. A visit to Retraction Watch will easily demonstrate this, though some papers are retracted for honest reasons.

But the article mentioned does highlight the problem with peer-review: It was developed in an era when researchers had more time - research money was not as highly contested, teaching loads were less, there was no pressure to publish - you published when you though the data were ready etc. There was plenty of time to pore over someone else's manuscript and give a thoughtful review of its content.

Some of the issues identified in the article linked above should have been caught by the associate editors of the journal -they are supposed to be choosing reviewers for the papers carefully before the paper is sent out: looking for connections between authors and reviewers, checking on the backgrounds of  reviewers and the like. The fact that they have not been doing this is another indication that people working with the papers are under pressure and feel they can't afford to take the time to look over the situation carefully.

So is peer-review doomed? I certainly hope not. The process has much merit to it, but it will struggle to survive until employers recognise the importance of peer-review to science progress. There needs to be a system in place where review of manuscripts is thought of as being of some importance - not as much as publishing a paper, but it needs to give some kudos to those who do it, and especially those who do it well.