Skip to main content
U.S. flag

An official website of the United States government

Reproducibility: Another bureaucratic speed bump?

John Williams
John WILLIAMS,
Health Scientist Administrator,
Division of Aging Biology (DAB)
.

So you thought your science was already rigorous and reproducible…

On October 5, 2015, NIH released new guidelines for implementing rigor and transparency in research project grants. Additional notices and guidelines followed. Applications for research grants and mentored career development awards submitted in 2016 must include information on scientific rigor and reproducibility. (Updates to institutional training grants, institutional career development awards, and individual fellowships will be forthcoming no sooner than 2017.)

If you’re a graduate student or postdoc, trying to reproduce someone’s published experiments as the starting point for a new line of investigation by following the Methods section in the paper, you might appreciate this effort. On the other hand, if you’re an established investigator, this effort may seem to verge on unnecessary bureaucracy added to an already brutal grant writing landscape. What’s going on?

What does the new policy cover?

The overall goal is to enhance a culture of transparency in biomedical research. Science moves most effectively when there’s a free exchange of ideas. You can find the current guidelines and more information about how to include them in your application on the Office of Extramural Research website.

The new grant application instructions under this policy fall into four main areas:

  1. The scientific premise of the proposed research – This is the research used to form the basis for the proposed research question(s). Applicants should describe the general strengths and weaknesses of the prior research they cite as crucial to support the application.
  2. Scientific rigor in the strict application of the scientific method to ensure robust and unbiased experimental design, methodology, analysis, interpretation and reporting of results.
  3. Consideration of relevant biological variables such as sex, age, weight, and underlying health conditions, which are often critical factors affecting health, not to mention experimental outcomes.
  4. Authentication of key biological and/or chemical resources, which can include, but are not limited to, cell lines, specialty chemicals, antibodies, and other biologics.

Reproducibility isn’t easy

I remember very well my frustration as a postdoc of trying to replicate someone’s exciting results in an effort to extend their observations, using their published Methods. It was nearly impossible.

A number of competing interests led to this situation. A key feature is that the continuing battle for precious space in a scientific journal has led to exceptional brevity in scientific writing, resulting in rather cryptic Methods sections. We learned quickly that if you really wanted to follow up on someone’s hot new data, you called the first author on the paper to make certain that you were following their experimental procedures exactly. It saved a lot of time and aggravation.

Scientists are an independent-minded bunch and so they often try to replicate the experiments and make certain assumptions about the procedures when the experimental designs are cryptic. Specific experimental designs are obviously critical and it isn’t safe to assume everybody would do the experiment the same way you would. Thus it isn’t surprising that reports eventually appeared in the media that some important experimental data was not reproducible…and here we are.

Scientific challenges

Non-standardized reagents are another problem. When I worked with serum and bovine serum albumin, using the same lot numbers was almost always critical for reproducibility. I will never forget the paper published in one of the top biomedical journals reporting that a new protein was “necessary and sufficient” for the differentiation of a particular cell type. This was a huge advance for the field in differentiating this cell type in culture. It so happens that the protein was “necessary” but the “sufficiency” depended on the lot of serum that was used (the protein alone was not sufficient as claimed). The fact that it was necessary to “find” specific lots of serum that would permit the differentiation in the presence of the protein was not included in the Methods or figure legends of the paper.

A lot of variability exists with antibodies, as well. Many supply companies sell antibodies which are not appropriate for all applications. And, remember that not all antibodies are created equal—even those for the same protein!

The new initiative’s prompt to include both sexes even in preclinical and basic research rings true here at NIA. Our Intervention Testing Program has found several unexpected differences between males and females of the four-way-cross mice they use. And the thought of age being taken seriously as a variable across NIH sure makes a lot of sense here, too. It would seem that these changes will eventually lead to changes in the scientific review culture (albeit slowly) because this will change the cost structure of grant applications.

What do you think?

So, are these guidelines well-intentioned folly? Or steps in a necessary direction? Ultimately the research community will decide that by how we respond to these new strictures. Are they an insufferable burden? Or are they a framework to make all research more transparent and generalizable? Let us know what you think by commenting below.

nia.nih.gov

An official website of the National Institutes of Health