Research and Funding
Inside NIA: A Blog for Researchers

The Approach criterion: why does it matter so much in peer review?

I am not surprised to see the high correlation between the approach criterion score and the final impact score. I have been on NIH study sections for the past 10 years or so, and I can say that the majority of applications come from qualified investigators in good environments. So there's not going to be a lot of variance in those scores to explain impact. Most of the applications also address a problem of at least reasonable significance, so there goes that variance. What separates a good application from a great one (and we all know that a grant has to be at least great to get funded these days) is usually the approach: how the investigators have operationalized their theoretical or practical question, whether they have attended to the important methodological issues (not the nit-picky ones, the BIG ones), and so on. I have also seen innovation influencing impact scores more and more, so I won't be surprised if that correlation gets larger over time.
The comments you cite in your examples are precisely what is wrong with the approach of many reviewers. Both reflect subjective rather than scientific judgments, i.e. not whether the approach represents good science and is likely to have an impact regardless of the experimental results, but whether it represents the reviewers' particular scientific approach. In an environment in which an unfavorable comment from a single reviewer, particularly if it is from a primary or secondary reviewer, and no one else on the study section has read the proposal, it is the kiss of death. It is also interesting that the correlation between innovation and impact is less than that of approach and impact, since the evaluation of the methodology is much easier than assessing the quality of the innovation. It represents the conservatism that has always been rampant in the peer review system but, because of current fiscal issues, is now much more likely to lead to innovative proposals falling below the payline or being triaged by reviewers who may not be particularly perceptive regarding the significance of the innovation.
I am now on the verge of retirement after 45 years in academia, during which I have been quite nicely funded by the NIH and remained so upto this year. I was also a chartered member of a study section and served on ad hoc basis in several others. During this time I have developed a perspective based on my impressions, which may, in a certain sense, be relevant to this discourse. I believe that the composition and personalities of the study section members are a significant if not a decisive factor in the outcome. Cronyism and factionalism are very prevalent. In my view, the weak link in NIH review system is the mechanism for the selection of members. Too much discretion is left to the SRAs, who quite often were never the main players in the scientific arena and thus lack gravitas. Very good scientists have often no hope of being funded while their "professional enemies" sit on a particular Study Section, especially if members are allowed to vote out of the range and the funding level is low. I am struggling to say it nicely but I have encountered some particularly nasty people on the study sections, who never liked anything unless it came from a crony. Enough!
As an early career person, there's a lot I don't know about the NIH review process but the comment by anonymous II especially caught my eye. It may be a silly question, but are the reviews themselves ever reviewed? What I mean is, does someone higher up the food chain check the reviews of study section members for outliers that might indicate bias/cronyism/factionalism?
Because each application is usually assigned to at least three reviewers, the purported bias/cronyism/factionalism would have to be rampant/pervasive and the reviewers in study sections in collusion for the charge to stand. I have been reviewing grant applications for the NIH continuously for 16 years, twice as a chartered member of a study section and countless times as an ad hoc, and in all of those years not a single time have I ever approached or been approached by another reviewer to discuss how an application should be scored. Almost invariably, the only time I would meet the others reviewers would be at the study section meeting, meaning that my ONLY interaction with the reviewers was at the meeting itself. When do we collude (these days a reviewer does not even know who the other reviewers who were assigned the same application(s) are)? The consequences of what I just stated should be obvious but I will spell them out: (a) Do reviewers themselves get reviewed? Yes, by each other, since no application is ever assigned to a single reviewer, so that each receives multiple independent evaluations, whose validity is discussed in the open by an even larger panel to arrive at the final decision that will determine the impact score; and (b) is there bias/cronyism/factionalism? An emphatic "no", based on my 16 years of experience as described above. The absence of cronyism and factionalism also supports the validity of the response to question (a) because it means that the reviewers can indeed review one another.
In my experience as a reviewer, most reviewers are familiar with the subject of given set of application in a study section, but they may not be experts in the subject. Therefore, it is critical that an application is assigned to an expert, who is in position to evaluate proposed approaches. When a reviewer is familiar with the proposed approaches, grant applications tend to do better. I agree with comments above that if reviewers have a bias towards an approach, grant is not likely to get a fundable score.