Today’s post was written by America Negrete. America is a senior Marketing major with a strong interest in Accounting. She works at two separate departments in the library and loves every bit!
October’s edition of Faculty Pub Night featured Ben Fitzpatrick, Professor of Mathematics. As a co-author of “Getting Personal with the ‘Reproducibility Crisis’: Interviews in the Animal Research Community,” Fitzpatrick joined students, staff, and faculty in the William H. Hannon Library’s Von der Ahe Suite to speak about the harsh realities of research in pharmaceuticals.
His publication details the increasing challenge that is research reproducibility. During his presentation, Fitzpatrick focused on two pharmaceutical companies, Bayer and Amgen. In 2011, Bayer attempted to reproduce 76 published studies. Only 14 were successfully reproduced. Similarly, Amgen revisited 53 studies. Of those, 6 were consistent with publications. This is a problem for pharmaceutical companies because research and development costs are consistently increasing, but drug discoveries are decreasing. Fitzpatrick and his co-authors, Elena Koustova and Yun Wang, interviewed 131 scientists, officials, and administrators in academia, government, industry, and private foundations in order to gain more insight on the reproducibility crisis. He continues to describe Type I and II Errors as well as p-values.
A p-value is an index used to assess the strength of evidence, where a smaller value indicates stronger evidence. A p-value of .05 is generally seen as acceptable and any study that produces a higher value is “not publishable.” Unfortunately, Fitzpatrick continues, academic culture in regards to publication is highly toxic. The pressure to publish in research-focused institutions is extremely high. Because only positive results are publishable, the stress of crafting a “perfect” study is overwhelming.
To illustrate the pressure that researchers are placed under, Fitzpatrick comes back to discuss Amgen’s replication work. An interviewee from the company shared that an Amgen scientist met with a study author to discuss his/her publication. The Amgen scientist commented that the study had been repeated at least 50 times and the original publication’s results were never achieved. The study author replied, “We repeated [the study] 6 times. One worked, and that’s what we published.” This example illustrates two difficulties: one, to replicate studies and two, to produce publishable studies.
The general reactions among the audience were confusion and fear. Weren’t studies published in academic journals truthful and valuable? After the night’s revelations, many were no longer certain. At least two audience members questioned how consumers of information can verify validity and trustworthiness among peer reviewed journals. It’s not clear, says Fitzpatrick. He references John Oliver’s Last Week Tonight segment on scientific studies. Oliver discusses the multitude of studies that constantly contradict each other. Fitzpatrick’s final verdict: the fact that there is no clear answer, is off-putting.
To turn the conversation around, one patron posited that if the accepted p-value was raised to .06, then the amount of replicable studies would rise. However, Fitzpatrick countered by saying that the idea of “chasing a threshold” would not change. Researchers would begin to chase a different p-value, and the problem would persist.
To close, Fitzpatrick revealed that a majority of his interviewees only took one statistics class throughout their higher ed careers. Few felt that one class met their research-based needs. As a result, Fitzpatrick believes that one answer to the reproducibility crisis could be to rethink introductory statistics courses towards research needs.