What Does ‘Reproducibility’ Mean?

Drugmonkey does a fine deconstruction of the latest Amgen/Nature paper, so I won’t rehash the whole thing here (go read it; the comments really blow up the details of the replications). But a key point is best summarized by Drugmonkey:

…I have long suspected many of these alleged replication failures claimed by Amgen are of the lack-of-generalization variety. A drug company of course wishes they can take a single hot basic science result and turn that into a human medication. The fact this doesn’t work out is not a failure of the original paper to replicate. And ffs the overblown translational claims of your typical Glam authors doesn’t change this reality.

It raises some real questions about what the reproducibility crisis is about. While there are some obvious cases, such as the ESP studies, where this is a statistical assessment issue, that seems to be conflated with ‘generalizability’–that is, if I change my study system slightly (different organisms or strains, slightly different methods, etc.) and I don’t observe the ‘same’ result (or a much weaker result), did the study fail to reproduce? Given real constraints of time and money, it’s often not feasible to replicate a previous experiment and conduct your own follow-on experiment as well.

I’ve spent much of my career in an open-data field–and the grunt work of my day job, in part, consists of trying to replicate other people’s results using their data (pipeline development, etc.). Even using the same data, slightly different processing and analysis steps can yield different results (though what ‘different’ means is open to discussion). Reproducibility isn’t a trivial thing to assess, since the real issue is how much irreproducibility is tolerable.

What bothers me about the reproducibility discussion is that, for the most part, we’re not discussing what reproducibility would and should mean, but rather it’s more along the lines of ALL TEH SCIENTISMZ ARE BROKEN! Which makes for great pundity thinky-pieces, but doesn’t really help us do science within the real-world constraints that scientists experience.

I will now go outside and yell at clouds.

This entry was posted in Statistics. Bookmark the permalink.