How To Do Replication the Fair Way (You Get What You Deserve Report)

In a long, but very good article about replication in science, I like this suggestion the best (boldface mine):

Kahneman rightly notes that the methods sections of most articles reporting original research are insufficiently detailed to allow third parties to conduct precise replications. As a result, if the goal were to replicate what the original researcher actually did, initial communication with original authors to learn more about their methods would indeed often be necessary for a replication to be scientifically valid.

However, an alternative legitimate goal that does not require consultation with original authors is replicating what the authors reported. That, after all, is what has entered the public record as a claim about reality. If an original study reports that, “if X and Y are done, A occurs,” it will not do for its author to respond to a replicator’s failed attempt to produce A from X and Y by suddenly insisting that another condition, Z, is also critical to reproducing the effect. A good-faith attempt to replicate exactly what was published cannot be criticized as invalid.

Rather than compromise the independence of replications by requiring their authors to consult with original study authors, the scientific standards of the future should require original authors to make all materials necessary for a precise replication of their study publicly available upon publication. This is analogous to the bargain that lies at the heart of U.S. patent law.

Even though I hate supplemental sections, they are, sadly, here to stay. There is no reason based on page limitations why your methods can’t be described in excruciating detail. If someone doesn’t know that you have to wave incense over the PCR machine, they can hardly be faulted for doing failing to do so. It might also lead to better methods sections, and, perhaps, even an increased recognition of the limited domain of applicability of many experimental results. That is, the significance and effect of many results can be very twitchy and condition-dependent.

Well, a Mad Biologist can dream, can’t he?

This entry was posted in Statistics. Bookmark the permalink.