Skip to main content

Maybe we should call it “just-in-case” rather than “just-in-time”

NIA Blog Team Avatar

NIA Blog Team

So, you just received an automated email that asks you to submit “just-in-time” information for your application. Does that mean NIA is going to pay it? I wish! Unfortunately, that just-in-time request brings false hope to too many. Here’s some explanation of the just-in-time messages and our data on who gets funded. It might help you consider the priority of responding to a just-in-time request for information, if your application to NIA has a percentile score of 21 or poorer.

Why am I getting these automated just-in-time emails about my grant application?

In April 2012, NIH changed its policy on collecting just-in-time information for applications that are about to be awarded.

Until then, some delays arose in making awards. Key information on salaries, etc., was not available when needed. So, NIH staff reasonably thought if we require everyone who has some chance of award to submit the information after the summary statement is released, then it will eliminate these unnecessary delays.

Everyone with a score better than 40 is asked for extra information.

NIH chose a middling impact rating score (40) as the cut-off for requiring just-in-time information.  Now that may seem overly generous, but staff making the decision had to contend with study sections whose scoring varies widely and Institutes whose funding lines differ substantially. In that light, 40 seemed reasonable as the cut-off.

So how do I interpret that just-in-time request?

The good news is that some delays have been eliminated for awardees. The bad news is that the great majority of people who are required to submit just-in-time information will not receive awards.

Score-creep—the tendency for applications in more recent rounds to receive better scores in review than those in earlier rounds—has made the problem grow larger since the policy was introduced in 2012. And, though NIA does not have the toughest payline in NIH Town, most times we struggle to stay mid-range. So a score of 40 when converted to a percentile is almost always outside our funding line.

So, what do I do with these automated just-in-time requests for information?

A graph of probability of receiving funding, based on percentile scores. The x-axis presents the given percentiles, divided into six groups. These groups are labeled 1 to 11, 11 to 15, 16 to 20, 21 to 26, 26 to 30, and percentiles greater than 30. The y-axis measures the probability of getting funded, on a scale of 0 to 1, increasing in increments of 0.1. The graph demonstrates that overall, the higher the percentile, the higher the chance the application has of getting funded. Applications in the 1 to 11 percentile group are 100% likely to be funded, while applications in the 11 to 15 percentile group are funded have a 46% chance of being funded, applications in the 16 to 20 percentile group are funded have a 18% chance of being funded, applications in the 21 to 25 percentile group are funded have a 2% chance of being funded, applications in the 26 to 30 percentile group are funded have a 1% chance of being funded, and applications with a percentile score of 30 or poorer are likely to not receive funding.I show in the accompanying chart the likelihood of award from NIA in 2013 for a given percentile score. (This was produced with the assistance of Sanoj Suneja, a data scientist in my office whose analysis helps the NIA improve its grants process and administration.)

As we have maintained the same payline for four years now, the chart is similar for all recent years.

Use it to guide you in how high a priority you give to submitting the just-in-time information.

As you can see from the curve, if your application has a percentile of 21 or poorer there’s very little chance of an award. In those cases, I recommend giving a very low priority to sending just-in-time information when the application’s percentile is in that range—that would be about the priority you give to cleaning out a certain file drawer left from the pre-electronic age. If the very unlikely happens and we do prepare to make an award—yes, it can happen—you will receive other signals from us to submit the just-in-time information.

If the percentile is 20 or better, a quick email to your program officer will give you the best sense of what priority to give to this task.

When it’s close, which grant applications get funding?

Looking at the part of the chart where the slope is steep, you might also wonder what distinguishes those applications that get funding from those that don’t.

That is where our policies on new and early stage investigators have an effect. Other priorities at NIA—priorities that you can find in our Strategic Directions document—play some part in the decision, too. We also think about program balance (”We need to provide more support to that neglected research question”). And, in the future, the increased national focus on Alzheimer’s disease will carry more weight in shifting an application to “paid” status in that snowboard-slope region.

The just-in-time policy is one of these good ideas that led to unfortunate side effects. You can be saved some unnecessary work by sensibly interpreting the percentile score that your application receives. Meanwhile at NIA we are working with NIH to make the call for just-in-time information a little more tailored to those applications that are in some danger of funding.

So, do you have other questions about those just-in-time emails? Let me know by commenting below.


Read Next:

New resubmission policy