Every few months, some enterprising writer stumbles across various studies that demonstrate how much of the biological literature can not be replicated–at which point, this is reported as ‘news.’ A favorite study is Bayer’s (the company) attempt to replicate 67 drug target validation studies and could only do so for fourteen studies–that’s twenty percent for those of you keeping score at home (it’s World Series season…). Anyway, I’ve written about this problem before, so I won’t add to this subgenre (though, being an old-school liberal, I prefer to view these popular articles as a WPA-like program for writers).

Instead, I want to figure out what the NIH think about this. If most of the work you’re funding can’t be replicated, that is problematic. As some asshole with a blog put it (boldface added):

…the funding system is not designed to collect the large samples we need to do the science correctly. To seriously address this issue would require fewer and larger projects (even if we increased funding). For those who argue that this amounts to ‘picking winners and losers’, it’s worth remembering that we’re not picking a whole lot of winners right now.

I could handle twenty or thirty percent–mistakes are made and science is hard. But when two-thirds to four-fifths of what you’re funding doesn’t pass the reproducibility test, you have problems. Somehow I don’t think the Gates Foundation would be especially thrilled to discover that most of their projects are failing–when they thought they were working.

It also makes me wonder about the entire grant review process. At least some of the funded irreproducible work was strongly endorsed by reviewers (that’s what finishing in the top percentiles means), yet these same experts–and for the record, the Mad Biologist has occasionally been one of those experts–supported what would ultimately be irreproducible work.

One criterion that’s often used in review is publication, both simply having published and having published in the glamour magz. If many of these high profile publications are either irreproducible or have ‘only’ overstated their findings, then a key element in the review process is seriously damaged, if not fundamentally flawed.

While the NIH has considered using independent labs for verification of (some) drug trials, it’s not clear what, if anything, the NIH intend to do about the majority of ‘basic research’ studies (as opposed to clinical drug trials). Put another way, most NIH studies won’t be replicated by independent labs.

I know some will argue that other labs will ultimately discover that the work can’t be repeated (That’s How Science Works!), but prestigous labs apparently can publish problematic results and not be called on it at all, so I’m not convinced this will operate in many instances.

Someone might want to push the NIH on this, and hard.