To resubmit or not?
NIH announced a change in resubmission policy in April. Following the policy change, lots of attention focused on the application count. The research community worried that NIH would see a giant surge in applications. Will a large number of additional applications overwhelm the application review process? Will success rates shrink to the grim, low-single-digit numbers not seen since pre-ARRA days? We have to wait to find out, and there are a few sweaty palms round here!
NIH policy: what changed?
Applications will no longer be examined for similarity to previous applications from the same scientist. A researcher whose application was not funded can now submit the exact same proposal text for consideration as a new application (though, of course, we do not advise this).
This blog post covers a different feature of the April policy change: how investigators can make decisions about grant applications that are not funded the first time they are submitted for consideration. Sanoj Suneja, a data scientist in my office, contributed to this post.
If you’re not familiar with the lingo, A0 is the first submission of an application, while A1 is a resubmission of that same application, after some deeply considered changes.
With the policy change, investigators now have a real choice after an A0 grant application is not funded.
- Should the application be revised and resubmitted as an A1 application?
- Or, should it be reworked and submitted as a new, A0 application?
The fiscal year at NIH and NIA
Like the rest of the federal government, our fiscal year runs from October through September. For example, fiscal year 2012 is October 2011 through September 2012.
To answer these questions, we dug into how well NIA-assigned A1 applications submitted in 2012 and 2013 did in review and award. We hoped to gain insight that might help inform your decisions.
What do our data say?
The bar chart below shows probabilities of award for A0 and A1 R01 applications assigned to NIA in fiscal years 2012 and 2013.
The chart separates new investigators from established investigators submitting Type 1 applications, and from Type 2 applications. You probably already know that a Type 1 is a new application, while a Type 2 is an application for continued funding of an existing grant. We call those Type 2 applications competing renewals.
It is no surprise, really, that the only applications to have a decent chance of success on first submission are competing renewals, the Type 2 applications. New applications from both new and established investigators had single digit success rates on initial submission.
New investigators are more likely to get grants when they revise and resubmit unfunded applications.
The surprising result for us happened on the A1 submission. It turns out that new investigator A1 submissions had a higher chance of award than established investigator A1 submissions for new applications. (New investigators revise and resubmit about 29% of their unfunded applications that were assigned to NIA. Established investigators are a bit more likely to resubmit, but the proportion is generally similar.)
Is this an artifact of the scoring advantage given to new investigators? It’s not.
It’s true that we gave a three (for new investigators who are not early stage investigators) or five (for early stage investigators) point advantage in the funding line to new investigators. So what’s the big deal? Surely that influences whether their A1 applications are awarded?
Well, we also looked at the percentile scores that new-investigator and established-investigator Type 1 applications received and found the same effect. New investigator resubmitted applications are more likely to obtain scores within the first 11 percentile points—the range that’s most likely to be funded—than established investigator resubmissions. So even when we compare within the same percentile range, we still see the new investigator advantage.
Bear in mind that the scores of the prior A0 applications from the new investigators were worse on average than the scores for the established investigator A0 applications. So, the A1 resubmissions from new investigators really did show a substantially larger improvement in score than the resubmissions from established investigators.
So, should I work on an A1 or an A0?
The takeaway message is that new investigators are somewhat more advised to work on a resubmission application than established investigators submitting Type 1 applications.
Of course, most critical still is reading the summary statement, discussing it with colleagues and with your program officer, and working out whether you can mount a point-by-point answer to the critical comments from the review.
Why do resubmitted grant applications from new investigators do so well?
We have some ideas, and you probably have some thoughts as well. Let us know why you think this is happening by commenting below.
- Learning the study section: New investigator resubmissions improve their science and also present the application in a more study-section friendly way. Established investigator resubmissions might improve only their science because their A0 was already study-section friendly.
- Contrast effect: New investigator R01 applications are reviewed together, separate from established investigator applications. Resubmissions for new investigators might contrast sharply with weaker A0 new investigator submissions. On the other hand, established investigator resubmissions do not contrast so sharply with the stronger established investigator A0 submissions. So, a contrast effect works in favor of new investigators’ resubmissions.
Of course you may have other ideas and we welcome your thoughts on it.