Buried within our NIA 2013 funding policy is the apparently shocking statement that our payline for NIA-reviewed research grant applications is 13. When the top score is 10 and the lowest possible score is 90, then a 13 is little short of perfect. We worry about this too. We know that we are leaving some truly outstanding work unpaid. How did this happen?
Which applications do we mean?
First, some important background: NIA-reviewed research grant applications include P01, U01, U19, and U34 grant applications. (Don’t understand what these alphanumeric codes mean? Better get yourself to the Types of Grant Programs tutorial without delay.) Applications for these kinds of grants are always reviewed by NIA study sections, rather than the more well-known review groups at the NIH’s Center for Scientific Review. Learn more about chartered review committees and special emphasis panels at the NIA.
What’s the difference in payline between NIA-reviewed and CSR-reviewed applications?
The payline for NIA-reviewed research grant applications is 13, out of a possible range of 10 (best) to 90 (worst). This means that NIA-reviewed applications with a score of 13 and below may likely be funded. Many CSR-reviewed R01 and R21 applications funded by us have scores worse than 13 (that is, higher than 13). The priority scores climb as high as 30! You might ask why CSR-reviewed research is apparently so privileged.
We focus on estimated success rate.
The key metric here is estimated success rate, not payline. And estimated success rate tells a very different story. Our estimated success rate for all NIA-reviewed applications this year is 18 to 23%. For program project applications it is in the range of 23 to 28%. You might be surprised, but this is really the case, even with a payline of 13! This means that a large percentage of applications reviewed at NIA receive nearly perfect scores, and we plan to fund a fifth to a quarter of applications. By comparison the success rate for CSR-reviewed research will be in the range of 14 to 17%. We can almost wonder why NIA-reviewed research is so privileged.
Sally Rockey did a great post earlier this year about the relationship of paylines and success rates. If you haven’t seen it already, please do check it out.
Now we can identify how this happened.
Our NIA-chosen reviewers are giving better scores on average to their assigned applications than CSR-chosen reviewers are giving to the NIA applications that they are assigned.
And why is that? Perhaps NIA-chosen reviewers have a finer appreciation for aging research than do CSR-chosen reviewers. Or maybe, research reviewed at NIA is of higher quality than aging research reviewed at CSR. Or it could be that reviewers chosen by NIA are more generous people than are CSR-chosen reviewers. Maybe all are true to some degree, or maybe there’s another factor we haven’t identified. If you have thoughts about what it might be, please let me know by commenting below.
How are we responding?
No matter the reason for the difference in scores between NIA-reviewed and CSR-reviewed applications, we need to pay attention both to overall success rates and to relative paylines when balancing the portfolio of NIA-reviewed and CSR-reviewed research. Because we are looking at success rates as well, at times, the NIA-reviewed payline can appear miserly.
Do you have follow-up questions? Please submit them as comments, below. I will read all comments, and may do a post in response to your thoughts.