Chartered change: Reorganization of NIA review committees
I am proud to direct NIA’s Scientific Review Branch (SRB). My colleagues and I strive to run fair and impartial reviews across a wide spectrum (and large volume) of research applications including training/career development, contracts, research centers, program projects, scientific meetings, and responses to NIA requests for applications. We also are tasked with assuring NIA is adhering to NIH and federal policies and practices.
Since we are all scientists at heart, we also review our own processes regularly, analyzing data to see what the trends can tell us. Recently, the SRB team put together an all-star planning committee made up of NIA leaders, scientific review officers, and subject matter experts to take a closer look at how our current distribution of applications to the standing committees was working and whether there were ways we could improve.
Follow the data
The present NIA standing committees (abbreviated as NIA-B, -C, -N, -S, and -T) review career development award applications combining different activity codes under five criteria:
- Candidate career development plan
- Career goals and objectives
- Research strategy
- Mentors, co-mentors, consultants, and collaborators
- Environmental and institutional commitment to the candidate
In our review, we noticed some inconsistencies in the scoring patterns among the five standing review committees. While some placed a strong emphasis on the proposed research strategy, others concentrated on either the candidate, the mentors, or the institution, disproportionately affecting the overall impact score. This difference in focus generated inconsistent scoring for a given mechanism (e.g., K99, K01) between committees, raising questions of fairness to the review process and consequently, the candidates.
Evolving our committee structure
Following extensive internal discussions and input from NIA leadership, we have decided to restructure and reorganize our standing committees to better ensure equitable review for all applications. This change will take effect starting with the receipt date of Feb. 12, 2021.
To better maintain fairness and consistency in evaluation, the four revised committees will be based on the award mechanism, instead of the scientific focus area. The roster for each committee will include reviewers with expertise encompassing the areas of science within the NIA’s mission and with knowledge and experience in training and mentoring. The new committee structures are detailed below:
- NIA – CD1: Career development facilitating the transition to independence
- Focus: K99/R00
- NIA – CD2: Career development for early career investigators
- Focus: K01, K25
- NIA – CD3: Career development for clinicians/health professionals
- Focus: K08, K23, K24
- NIA – CD4: Career development for established investigators and conference grants
- Focus: K02, K07, K18 + R13
Uniform evaluations: Regardless of mechanism
We strongly believe that these proposed changes will improve uniformity of evaluation. The move to mechanism-focused panels will also reduce burden for reviewers since they will be able to focus on a single mechanism (or a set of related ones) rather than learning and adjusting to the review of multiple mechanisms within each study section.
For applicants, this reorganization will provide a clear pathway to which panel an application will be assigned. We also hope this change will give us more opportunities to recruit and train junior faculty from diverse backgrounds to be reviewers.
Moving forward, we will continue to evaluate the success of this reorganization annually and make changes as needed. We understand this change is likely to generate questions, and you can contact us directly or post your comments or concerns below. We look forward to improving our service for you and reading your next round of applications!
Comments
Thank you for trying something new with the restructuring of the career development application review committees. It is disheartening for reviewers to give an application a high recommendation only because of the mentor's reputation rather than the merit of the applicant and science proposed.
Sometimes it is hard to change the thinking and outlook but hopefully with the new structure we can overcome the past discrepancies and achieve better calibration.
I hope this new structure will help in creating a true and fair assessment of the applications and decrease the chronic bias and historical culture that only applications from well-established programs and investigators with a long-lasting reputation are deserving of such opportunity. Perhaps, the first tier review should be blinded and the scientific/approach should be the first component to be evaluated. In addition, an evaluation process, that includes the applicant's experience with the process, mentors, supervisor, and environment should be considered.
Dear Dr. Heyn,
Thanks for your comment and also for your positive assessment of the new changes we have just implemented; we will continuously carefully monitor the behavior and attitude of the reviewers and make changes as needed. The evaluation and progress of our grantees is constantly being monitored through annual progress reports and at sessions devoted to at scientific meetings. Additionally, we are looking for input from concerned people like you and others for new innovative ideas to improve the system.
Ramesh Vemuri
I have a question about resubmissions of career development grant applications. If a proposal was originally submitted in 2020 (and reviewed under the old committee structure), and will be resubmitted after 2/12/21, what committee will it go to?
Effective with receipt dates on and after February 12, 2021, all K applications will be assigned based on the new committee structure. So, a K99 application will be assigned to NIA-CD1, a K01 application to NIA-CD2, and so forth. The existing members will be distributed among the new committees and they will have access to your previous summary statement.
Dear Ramesh:
This is a great idea! I like the restructuring such that each section review specific parts of the career path. I believe you will get more consistent results with this method.
Charlie
Thanks Charlie for your support and the changes were made after long and careful consideration of pros and cons. Hopefully we will achieve uniformity and consistency across a given mechanism.
I noticed that for years certain study section have minimal change of rosters and they typically include the same people rotating between different study sections. More diversity and turnover should be in one year, as resubmission of the same application receive the same bias from the committee that reviewed it in the same year. No members should return to the same study section in less than 12 months.
Thanks
Standing study section typically members sign on to a 4-year term and commit to attend 3 review meetings per year. These committee members provide consistency (e.g., knowledge of mechanisms, score calibration) across multiple meetings. Ad hoc members are recruited based on the need for scientific expertise and vary from meeting to meeting.
While there is a need to balance consistency and turnover, NIA (and NIH overall) appreciate the importance of bringing in new reviewers and avoiding the overuse of existing reviewers. We also understand your concern and try not re-recruit retired members for a period of time. Having said that, we are always looking for new cadre of diverse reviewers, so if you are interested, please register at: https://www.nia.nih.gov/research/scientific-review/want-be-reviewer-nia.
This is a good idea indeed and many NIH panels might benefit from
a close look given the long tenure of panel members and the difficulty of assigning appropriate reviewers to proposals as research fields evolve. I had a recent experience with the R21 panel where a reviewer provided no feedback , generic comments, and proposal sinking scores while the two other reviewers were highly positive. The PO discouraged any appeals and said "bad reviewers can happen" but its very difficult to do anything about it. The advice of just re-submit and point out the disconnected reviewer of course just puts the PI in another 4-6 month funding cycle which can of course be disasterous to the research ever seeing the light of day. So any chance the R21, R01 panels in NIA are also being reorganized?
Thanks for your input and we are really sorry that your recent experience was not conducive. Despite continuously educating the reviewers on the importance of the need to write detailed critiques accompanied by matching criterion scores, some do not always adhere to these instructions. We at NIA even allocate additional two to three days post review meetings for editing their comments and also constantly remind the reviewers to give appropriate feedback to the applicants.
The R21 and R01 applications reviewed by NIA are usually in response to the RFAs and we always strive to improve the quality of reviews with input and suggestions from investigators like you. We appreciate your comments.
Do you anticipate any variation with resubmissions? Mine went in clinical the first time and the resubmission is now assigned to AGCD-2. Thank you in advance.
Dr. Schiaffino, we have redistributed the reviewers from the old committees to the new committees based on the expertise needed and familiarity with a given mechanism. Our SROs are recruiting additional ad hoc members to supplement the existing panel to fill gaps, if any. Be assured that we will do our best to make sure that each application gets a fair and appropriate review.
As a semi-frequent peer reviewer, it continues to trouble me and others that MANY peer reviewers refuse to use the full range of scores (1-9). This creates the typical problem of grade inflation and is a disservice to truly strong applications and the overall research workforce. I urge you to consider forced ranking within each reviewer to create a spread, or not including reviewers who consistently only use a very narrow scoring range.
Dear Dr. Wu,
First of all let me thank you for your service on behalf of NIA and NIH, given your interest in improving the review process and the system hope you will become a regular member rather than a semi-frequent reviewer.
Secondly, the skewing of scores has been a vexing problem for all and we at NIA confronted this situation few years ago when we could not fund an application with a score of 15. As a result we have instituted three changes which has resulted in a greater distribution/spread of scores:
1. Two stage review where the first stage reviewers evaluate the application, score different components and the second level reviewers read all the critiques (see the forest rather than trees) and give the overall score;
2. We ask the reviewers recalibrate their initial scores following the initial triage; and
3, Orient the reviewers (even the experienced) and the need to spread the scores (not just for formality sake), emphasize that their critiques (strengths and weaknesses) should match their overall score.
We will continue to monitor the whole review process as a continued process, make changes as needed and any input from seasoned reviewers like you are always welcome.
Add new comment