Skip to main content

Chartered change: Reorganization of NIA review committees

Ramesh Vemuri
Ramesh VEMURI,
Chief, Scientiifc Review Branch,
Scientific Review Branch (SRB)
.

I am proud to direct NIA’s Scientific Review Branch (SRB). My colleagues and I strive to run fair and impartial reviews across a wide spectrum (and large volume) of research applications including training/career development, contracts, research centers, program projects, scientific meetings, and responses to NIA requests for applications. We also are tasked with assuring NIA is adhering to NIH and federal policies and practices.

Since we are all scientists at heart, we also review our own processes regularly, analyzing data to see what the trends can tell us. Recently, the SRB team put together an all-star planning committee made up of NIA leaders, scientific review officers, and subject matter experts to take a closer look at how our current distribution of applications to the standing committees was working and whether there were ways we could improve.

Follow the data

The present NIA standing committees (abbreviated as NIA-B, -C, -N, -S, and -T) review career development award applications combining different activity codes under five criteria:

  • Candidate career development plan
  • Career goals and objectives
  • Research strategy
  • Mentors, co-mentors, consultants, and collaborators
  • Environmental and institutional commitment to the candidate

In our review, we noticed some inconsistencies in the scoring patterns among the five standing review committees. While some placed a strong emphasis on the proposed research strategy, others concentrated on either the candidate, the mentors, or the institution, disproportionately affecting the overall impact score. This difference in focus generated inconsistent scoring for a given mechanism (e.g., K99, K01) between committees, raising questions of fairness to the review process and consequently, the candidates.

Evolving our committee structure

Following extensive internal discussions and input from NIA leadership, we have decided to restructure and reorganize our standing committees to better ensure equitable review for all applications. This change will take effect starting with the receipt date of Feb. 12, 2021.

To better maintain fairness and consistency in evaluation, the four revised committees will be based on the award mechanism, instead of the scientific focus area. The roster for each committee will include reviewers with expertise encompassing the areas of science within the NIA’s mission and with knowledge and experience in training and mentoring. The new committee structures are detailed below:

  • NIA – CD1: Career development facilitating the transition to independence
    • Focus: K99/R00
  • NIA – CD2: Career development for early career investigators
    • Focus: K01, K25
  • NIA – CD3: Career development for clinicians/health professionals
    • Focus: K08, K23, K24
  • NIA – CD4: Career development for established investigators and conference grants
    • Focus: K02, K07, K18 + R13

Uniform evaluations: Regardless of mechanism

We strongly believe that these proposed changes will improve uniformity of evaluation. The move to mechanism-focused panels will also reduce burden for reviewers since they will be able to focus on a single mechanism (or a set of related ones) rather than learning and adjusting to the review of multiple mechanisms within each study section.

For applicants, this reorganization will provide a clear pathway to which panel an application will be assigned. We also hope this change will give us more opportunities to recruit and train junior faculty from diverse backgrounds to be reviewers.

Moving forward, we will continue to evaluate the success of this reorganization annually and make changes as needed. We understand this change is likely to generate questions, and you can contact us directly or post your comments or concerns below. We look forward to improving our service for you and reading your next round of applications!

Comments

Submitted by Catherine Jankowski on January 27, 2021

Thank you for trying something new with the restructuring of the career development application review committees. It is disheartening for reviewers to give an application a high recommendation only because of the mentor's reputation rather than the merit of the applicant and science proposed.

Sometimes it is hard to change the thinking and outlook but hopefully with the new structure we can overcome the past discrepancies and achieve better calibration.

Submitted by Douglas Barthold on January 27, 2021

I have a question about resubmissions of career development grant applications. If a proposal was originally submitted in 2020 (and reviewed under the old committee structure), and will be resubmitted after 2/12/21, what committee will it go to?

Submitted by Ramesh Vemuri on January 29, 2021

In reply to by Douglas Barthold

Effective with receipt dates on and after February 12, 2021, all K applications will be assigned based on the new committee structure. So, a K99 application will be assigned to NIA-CD1, a K01 application to NIA-CD2, and so forth.  The existing members will be distributed among the new committees and they will have access to your previous summary statement.

Submitted by Charles DeCarli on January 27, 2021

Dear Ramesh:

This is a great idea! I like the restructuring such that each section review specific parts of the career path. I believe you will get more consistent results with this method.

Charlie

Thanks Charlie for your support and the changes were made after long and careful consideration of pros and cons. Hopefully we will achieve uniformity and consistency across a given mechanism.

Submitted by Charbel Moussa on January 27, 2021

I noticed that for years certain study section have minimal change of rosters and they typically include the same people rotating between different study sections. More diversity and turnover should be in one year, as resubmission of the same application receive the same bias from the committee that reviewed it in the same year. No members should return to the same study section in less than 12 months.
Thanks

Standing study section typically members sign on to a 4-year term and commit to attend 3 review meetings per year. These committee members provide consistency (e.g., knowledge of mechanisms, score calibration) across multiple meetings. Ad hoc members are recruited based on the need for scientific expertise and vary from meeting to meeting.

While there is a need to balance consistency and turnover, NIA (and NIH overall) appreciate the importance of bringing in new reviewers and avoiding the overuse of existing reviewers. We also understand your concern and try not re-recruit retired members for a period of time. Having said that, we are always looking for new cadre of diverse reviewers, so if you are interested, please register at: https://www.nia.nih.gov/research/scientific-review/want-be-reviewer-nia.

Submitted by Stephen J Hanson on February 06, 2021

This is a good idea indeed and many NIH panels might benefit from
a close look given the long tenure of panel members and the difficulty of assigning appropriate reviewers to proposals as research fields evolve. I had a recent experience with the R21 panel where a reviewer provided no feedback , generic comments, and proposal sinking scores while the two other reviewers were highly positive. The PO discouraged any appeals and said "bad reviewers can happen" but its very difficult to do anything about it. The advice of just re-submit and point out the disconnected reviewer of course just puts the PI in another 4-6 month funding cycle which can of course be disasterous to the research ever seeing the light of day. So any chance the R21, R01 panels in NIA are also being reorganized?

Submitted by Ramesh Vemuri on February 08, 2021

In reply to by Stephen J Hanson

Thanks for your input and we are really sorry that your recent experience was not conducive. Despite continuously educating the reviewers on the importance of the need to write detailed critiques accompanied by matching criterion scores, some do not always adhere to these instructions. We at NIA even allocate additional two to three days post review meetings for editing their comments and also constantly remind the reviewers to give appropriate feedback to the applicants.  

The R21 and R01 applications reviewed by NIA are usually in response to the RFAs and we always strive to improve the quality of reviews with input and suggestions from investigators like you. We appreciate your comments.  

Add new comment

Your e-mail address will not be posted.
About text formats

Plain text

  • Allowed HTML tags: <p> <br>
  • No HTML tags allowed.
  • Lines and paragraphs break automatically.
  • Web page addresses and email addresses turn into links automatically.