A while ago, I wrote, “Someday, a science reporter is going to hybridize with an economics reporter and then the topic of how science is funded will actually be covered accurately. Until then, you’re stuck with the Mad Biologist.” Well, I don’t know if the hybridization experiment has been successful, but a Nature news article by Kendall Powell describes the grant selection process very accurately.
Before I get to the article, it’s not anything shocking or especially revealing to most scientists, but my hope is that journalists (and members of the chattering class) will read it and realize that how funding decisions are made is very important. Because the basic argument in the article is one that I’ve made before:
It’s analogous to college admissions at highly selective institutions (e.g., 5-15% acceptance rates). Most of these schools would be able to accept an entire class, get rid of it, take the next ‘class’ down, get rid that, and take the third cohort…and not miss a beat. And most will admit that if they went an additional class down, the drop off would be very little*. Ultimately, admissions officers have to come up with reasons to disqualify qualified students.
A similar phenomenon applies to NIH proposals. With the current acceptance rate of around twelve percent, roughly two thirds of perfectly fine proposals wind up being rejected. By “perfectly fine”, I mean proposals that have a reasonable chance of working out, and, if they did, would be interesting to other scientists in that area (i.e., if you heard a seminar on the work, you would conclude it’s solid work). I think about another twenty percent of proposals are basically one step away from moving into this category.
I’ve always though that proposals are like students applying to a ‘highly selective’ college: you kick the bottom two-thirds, there is a small number of really qualified students that you obviously want, and the rest are pretty interchangeable (not that you want to tell the
You get the idea. But the article is very useful in the examples it provides:
“It seems pretty pedestrian,” says the committee chairman, referring to the first application on the list. The applicant wants to investigate the molecular signals that could shut down runaway cell division in a particularly deadly cancer — but much of this pathway has already been worked out in other cell types. “This is good solid work,” argues another reviewer, slightly exasperated. “Not everything has to be a bright, shiny idea. Valuable information will come out of it. The innovation is less than in other grants, but I think the other aspects make up for that.”
What people need to understand is that, in an environment, where you have to come up with reasons to disqualify good proposals, the reasons become very arbitrary:
The deadly cancer application, which has been submitted twice before, is on its last chance according to ACS rules. “Every time, [this proposal] has been ranked as ‘outstanding’. It’s now or never for this one,” says one panel member. “The science is strong, but there is this issue of novelty that seems to be dogging this grant,” says another.
Its rival is a first-time submission, with the aim of studying the downstream events in a signalling network important in numerous cancers. It is technically superb, the reviewers agree, but the committee chairman, who reviewed it, has concerns about the applicant’s productivity. “This is well conceived, nicely written and, by the end of it, it’s really great science,” he says. “But this investigator had an extended postdoc and she had very few first-author publications.”
“The publication rate out of her postdoctoral laboratory is slower than most other labs,” points out another panel member. “The stories that come out [of that lab] are very big. That rate of publication is not unusual.”
Essentially, the argument is over the ‘flavor’ of a proposal (MAH PROPOSAL HAZ A FLAVOR!) versus concerns that a junior person hasn’t published enough. The ‘boring’ proposal has been rated ‘excellent’ three times. How weak could the science really be? (that, or the reviewers suck ass). The ‘unproductive’ proposer hasn’t published enough during her post-doc (which, I thought, was supposed to be, in part, about training–there has to be some slack granted; hell, maybe she had, you know, children or something stupid and unproductive like that…).
What I hope the ‘laity’ realize is that much of science–as well as scientists’ careers and opportunity to do research–are often determined by trivial things and weird panel personal dynamics*. Anyway, the article is very well done, and I hope this issue gets covered more–and maybe that will lead to actually rigorous scientific studies of funding too…
*I look back on one proposal in particular that fell right below the threshold and wonder how different my career would be if I had received it (there’s a lot of backstory not worth going into here).