Maybe We Should Give Up on Peer Review and Just Dump Everything in ArXiv

I kid, but consider this abstract, “Peer-review practices of psychological journals: The fate of published articles, submitted again”, that came trundling across the transom (boldface mine):

A growing interest in and concern about the adequacy and fairness of modern peer-review practices in publication and funding are apparent across a wide range of scientific disciplines. Although questions about reliability, accountability, reviewer bias, and competence have been raised, there has been very little direct research on these variables.

The present investigation was an attempt to study the peer-review process directly, in the natural setting of actual journal referee evaluations of submitted manuscripts. As test materials we selected 12 already published research articles by investigators from prestigious and highly productive American psychology departments, one article from each of 12 highly regarded and widely read American psychology journals with high rejection rates (80%) and nonblind refereeing practices.

With fictitious names and institutions substituted for the original ones (e.g., Tri-Valley Center for Human Potential), the altered manuscripts were formally resubmitted to the journals that had originally refereed and published them 18 to 32 months earlier. Of the sample of 38 editors and reviewers, only three (8%) detected the resubmissions. This result allowed nine of the 12 articles to continue through the review process to receive an actual evaluation: eight of the nine were rejected. Sixteen of the 18 referees (89%) recommended against publication and the editors concurred. The grounds for rejection were in many cases described as “serious methodological flaws.” A number of possible interpretations of these data are reviewed and evaluated.

Let’s leave aside the complete breakdown of any plagarism detection systems. If I did the probability theory right, the rejection of previously accepted papers is indistinguishable from the editors deciding to randomly accept papers with a twenty percent acceptance rate (with an acceptance rate of 20%, the probability of rejecting 9/9 papers is 13%, and the probability of rejecting 8/9 is 30%). I suppose the good news is that the study is too underpowered to detect a rejection rate definitively greater than would be expected randomly.

At this point, the only purpose of peer review that I can see is weeding out much of the utter bullshit (though even that fails occasionally).

Something Churchill said about the “worst of all possible systems” comes to mind…

Related post: If you think this is bad, read this post which describes how, at multiple journals, one reviewer claimed the results were impossible while the other claimed that ‘we already knew this.’

This entry was posted in Publishing, Statistics. Bookmark the permalink.

3 Responses to Maybe We Should Give Up on Peer Review and Just Dump Everything in ArXiv

  1. Min says:

    No wonder referees don’t get paid. 😉

    My point is actually that you get what you pay for, and referees should be well paid.

  2. Phillip Lord says:

    Peer review operates to get stuff out there slowly. I don’t see that it makes sense in this day and age. I publish my lab notebook generally without peer-review. I do sometimes get a second opinion when I want additional checking.

    Guess what? Scientists can get their own work checked to understand whether it is good enough for publication. What happens next is the real issue.

    So, yes, get rid of prepublication peer review. Post publication, well, that’s going to happen anyway.

  3. Jed Harris says:

    You should note that the linked article was published in 1982. Quite possibly things are worse now, but the time gap is significant. Checking for plagiarism is a lot easier and would prevent easy replication, for example.

Comments are closed.