Percentiling at NIA: sometimes we do. And sometimes we don't.
Why do some grant applications receive a percentile rank and others not?
Which NIA applications get a percentile rank?
Applications for an NIA grant that are reviewed at the NIH Center for Scientific Review, or CSR, get a percentile rank.
This percentile ranking orders these investigator-initiated research grant applications, with those with the best scores at the top of the list for funding. CSR-reviewed applications constitute the bulk of our funding. (For newbies, read about the NIH peer review process here.)
Applications for an NIA grant that are reviewed at the National Institute on Aging, through review panels organized by our NIA scientific review staff, do not receive percentile ranks.
If an application is reviewed by NIA, we give it a priority score, not a percentile. This includes the R01s and investigator-initiated research grant applications that are reviewed at NIA. But this seems inconsistent! Either this means that we at NIA are crazy (which many have claimed) or that the use of percentiling needs explaining. Well, today we’re going try to clear away all the clouds.
Why do we use percentiles for the NIA applications reviewed by CSR?
During each Council review round, we’re combining data from as many as 100 different committees from all over CSR in the same list.
Committees develop different patterns of behavior. Some concentrate their scoring at the good end of the range: all the applications are above average! Others make a conscientious effort to spread their scores throughout the range. And there are a few that hold to the maxim that nothing is perfect and so vote that way.
Percentiling, then, adjusts for these different patterns. Percentiles are based on three rounds of a particular panel’s behavior. Because the base is calculated from reviews by the same panel then the tendency of a panel to be generous or stingy with scoring is canceled by the percentile.
Percentiles allow the NIA to order applications reviewed by panels that behave very differently from each other and could otherwise have distorted a funding list.
Why don’t we do this for applications reviewed at NIA?
We tried this and it no longer works well.
In earlier days, we did percentile program project applications and other research grant applications reviewed at NIA. We used the trick of percentiling them against the average of all the CSR panels as there were too few applications in our own panels to percentile them against themselves. This method missed some of the advantages of percentiling but it did allow us to integrate NIA-reviewed applications with CSR-reviewed applications in the same list. And we saw that as an important advantage.
Then the scoring system changed.
We went from a score range of 100 to 500 to a range of 10 to 90. That increased the problem of “ties” and brought our percentiling solution crashing down on us.
And the challenges get even bigger when panels start handing out perfect scores of 10 like candy. Normally a score of 10 would be at the first percentile. But when a panel gives a lot of tens, then the percentile for a score of 10 drops to maybe the 5th or 6th percentile and a score of 11 would drop to the 10th or 12th percentile. That means an application with an almost perfect score would drop out of the funding range of many ICs. So CSR panels really have a disincentive to give out many 10s. (And there are other disincentives that deter CSR panels from giving out too many 10s.)
But, NIA panels aren’t in the same situation when it comes to penalties for assigning many 10s. All applications for which they assigned a score of 10 would receive a first percentile rating based on the CSR-averaged-percentile-score chart. And a score of 11 would be at the 2nd percentile and sometimes still at the 1st percentile.
That sounds like an esoteric problem—surely there are more pressing things to worry about? Until it happens. Then it becomes a real headache. Been there. Done that.
We tried other ways to percentile. To cut a long story short, they don’t work either.
Applications reviewed at NIA now get a priority score, not a percentile rank.
So, we chose to use the priority score itself to rank applications. We make a major effort to give different NIA review committees strong guidance on spreading scores to try to achieve consistent scoring across different committees. And we now have separate funding lines for NIA-reviewed and CSR-reviewed applications as we cannot integrate the two sets of applications.
So, the answer to why we percentile some applications and not others comes down to trying to keep an even playing field for science across quite different review environments.
Have I answered all of your questions about percentiling? Get in touch with me by commenting below.