A while ago, I wrote, “Someday, a science reporter is going to hybridize with an economics reporter and then the topic of how science is funded will actually be covered accurately. Until then, you’re stuck with the Mad Biologist.” Well, I don’t know if the hybridization experiment has been successful, but a Nature news article by Kendall Powell describes the grant selection process very accurately.
Before I get to the article, it’s not anything shocking or especially revealing to most scientists, but my hope is that journalists (and members of the chattering class) will read it and realize that how funding decisions are made is very important. Because the basic argument in the article is one that I’ve made before:
It’s analogous to college admissions at highly selective institutions (e.g., 5-15% acceptance rates). Most of these schools would be able to accept an entire class, get rid of it, take the next ‘class’ down, get rid that, and take the third cohort…and not miss a beat. And most will admit that if they went an additional class down, the drop off would be very little*. Ultimately, admissions officers have to come up with reasons to disqualify qualified students.
A similar phenomenon applies to NIH proposals. With the current acceptance rate of around twelve percent, roughly two thirds of perfectly fine proposals wind up being rejected. By “perfectly fine”, I mean proposals that have a reasonable chance of working out, and, if they did, would be interesting to other scientists in that area (i.e., if you heard a seminar on the work, you would conclude it’s solid work). I think about another twenty percent of proposals are basically one step away from moving into this category.
And:
I’ve always though that proposals are like students applying to a ‘highly selective’ college: you kick the bottom two-thirds, there is a small number of really qualified students that you obviously want, and the rest are pretty interchangeable (not that you want to tell the
customersstudents that…).
You get the idea. But the article is very useful in the examples it provides:
“It seems pretty pedestrian,” says the committee chairman, referring to the first application on the list. The applicant wants to investigate the molecular signals that could shut down runaway cell division in a particularly deadly cancer — but much of this pathway has already been worked out in other cell types. “This is good solid work,” argues another reviewer, slightly exasperated. “Not everything has to be a bright, shiny idea. Valuable information will come out of it. The innovation is less than in other grants, but I think the other aspects make up for that.”
What people need to understand is that, in an environment, where you have to come up with reasons to disqualify good proposals, the reasons become very arbitrary:
The deadly cancer application, which has been submitted twice before, is on its last chance according to ACS rules. “Every time, [this proposal] has been ranked as ‘outstanding’. It’s now or never for this one,” says one panel member. “The science is strong, but there is this issue of novelty that seems to be dogging this grant,” says another.
Its rival is a first-time submission, with the aim of studying the downstream events in a signalling network important in numerous cancers. It is technically superb, the reviewers agree, but the committee chairman, who reviewed it, has concerns about the applicant’s productivity. “This is well conceived, nicely written and, by the end of it, it’s really great science,” he says. “But this investigator had an extended postdoc and she had very few first-author publications.”
“The publication rate out of her postdoctoral laboratory is slower than most other labs,” points out another panel member. “The stories that come out [of that lab] are very big. That rate of publication is not unusual.”
Essentially, the argument is over the ‘flavor’ of a proposal (MAH PROPOSAL HAZ A FLAVOR!) versus concerns that a junior person hasn’t published enough. The ‘boring’ proposal has been rated ‘excellent’ three times. How weak could the science really be? (that, or the reviewers suck ass). The ‘unproductive’ proposer hasn’t published enough during her post-doc (which, I thought, was supposed to be, in part, about training–there has to be some slack granted; hell, maybe she had, you know, children or something stupid and unproductive like that…).
What I hope the ‘laity’ realize is that much of science–as well as scientists’ careers and opportunity to do research–are often determined by trivial things and weird panel personal dynamics*. Anyway, the article is very well done, and I hope this issue gets covered more–and maybe that will lead to actually rigorous scientific studies of funding too…
*I look back on one proposal in particular that fell right below the threshold and wonder how different my career would be if I had received it (there’s a lot of backstory not worth going into here).
mikethemadbiologist
Your little aside at the end (not that you should expand or elaborate if you don’t want to) is the most important part of this whole story, in a sense. We now have a system that for many people means that the funding game is the most important thing, by several orders of magnitude. Simultaneously, we have a system that, if we are being extremely charitable, rewards people at random (not literally, but within the pool of well crafted proposals, only 25-40% get funded, and the rest fail even though they are every bit as worthwhile). If we are being less charitable, it creates a system where who you know is MUCH more important than what know, and one where the system protects itself from outsiders almost by default. Let me suggest, humbly, that this is no way to run a scientific enterprise!
On another note, it is worth mentioning here that there is still a lot of room in the scientific community for scientists to get funded, even when the traditional RO1 route is difficult to impossible. It can be quite advantageous to work at a smaller, less prestigious institution…
This is, of course, true of almost everything. My BIL is an electrician, and his firm gets its bids chosen sometimes because they are lowest, but if they are in a critical fiscal window, they get chosen by what people happen to know about them or by something someone’s FIL said about a guy who worked for him 3 years ago for three months. The other electricians are good and responsible and bonded too – and sometimes it works out in my BIL’s favor and sometimes not.
This is true in writing – I just had a piece rejected from a magaizine because the editor was a Star Trek fan and I had a Star Trek reference he disapproved of in it. The reference was totally trivial, and everyone but this editor thought it was funny. The person whose piece did get accepted and I were emailing back and forth (we know one another well) and she told me she’s been dump from another magazine because she used the phrase “cannot help being” and the editor finds it uneuphonious.
This is true in hiring – I sat on a hiring committee recently that had three excelleng candidates. One of them had associations with an agency that has a high profile person that one of my colleagues dislikes – the person in question had no association with that person, came from another end of the field entirely. This did not prevent the other member of the hiring committee from identifying this as a potential reason not to hire that person, and in the end, none of us could see any reason not to hire all three of them, except that we are budgeted for only one. So we let that stand, stupid though it was.
The gist of this post really seems to be “scientific careers are just like every other career – they do not exist only on merit.” This is certainly true, but it seems weird that you’d have to say this specifically about science. Or maybe it isn’t weird.
Sharon
You are all so right! thnx!