Faced with confusing or contradictory COVID-19 health news, the public needs more help in understanding complex scientific studies. Michigan State University researchers say.
“Scientific studies, especially those in health and medicine, are often interesting — or at least relevant — to the general public. If results are conveyed accurately and thoughtfully, the hope is that they can inform heathy choices and behavior,” said Kenneth Frank, MSU Foundation professor of sociometrics. “The problem is that science isn’t certain. And it can be difficult to communicate the uncertainty of scientific results.”
A new paper authored by Frank and MSU colleagues Thomas Dietz and Lixin Zhang — as well as colleagues from University of Chicago, Arizona State University, the University of Connecticut, the University of Tennessee and Optinose Inc. — maps out a clearer and more effective way of expressing this uncertainty. The method is all the more relevant now, since the coronavirus pandemic has jump-started the rate at which scientific information is disseminated.
Frank said that a good example of the difficulty in communicating study results comes from a recent talk by Anthony Fauci, the director of the National Institute for Allergy and Infectious Diseases. Fauci relayed trial results from a potential COVID-19 therapy, remdesivir: “Remdesivir improved recovery from 15 days to 11 days with a P value of 0.001. … We would have normally waited several more days. The data may change, but the conclusion won’t.”
Some people may be familiar with the P- value as a way of expressing certainty or uncertainty in a study result, but Frank said that the concept is unintelligible to most.
“Even Fauci doesn’t have the language to talk about this kind of finding in easily relatable terms,” Frank said. “Many people can’t understand it. So, our goal is really to improve the conversation. We propose a novel way of framing the uncertainty of results in terms of patient outcomes, rather than P-values alone.”
The crux of the method is this: Results are defined as the number of patients who would have had to experience a different outcome for the study’s conclusion to change.
“In this way, the method generates statements such as ‘xx of the treatment cases that experienced improvement would have to have experienced no improvement to change the conclusion at conventional levels of statistical significance,’” Frank said.
Sometimes as little as one switch is necessary to shift the results from “significant” to “insignificant” or vice versa.
Take as an example an early randomized trial of hydroxychloroquine, a medication that has been tested as a potential treatment for Covid-19. Frank’s paper finds that if just one participant in the study had had a negative or neutral response rather than a positive one, it would shift the results from significant to insignificant.
“One switch is not a lot,” Frank said. “It means that an estimate is right on the borderline of what is typically considered conclusive-inconclusive. And the new scientific debate that has ensued about the hydroxychloroquine studies coming out reinforces that uncertainty.”
Frank and his team have already made the algorithm available to the public in the form of an app http://konfound-it.com that performs the calculation after the user inputs basic data found in most results tables; they are also sharing real-time updates on their blog.
Some researchers are already beginning to adopt the method and Frank’s hope is that more will begin to do so, adding the extra conceptualization to their papers as an addendum to the usual reporting of results.
“Communicating the uncertainty of new findings, especially now with Covid-19, in the relatable terms of patients’ experiences is critical for informing public policy and clinical practice,” Frank said. “It’s more accessible to clinicians, researchers and the public. The underscore for me is that we’re talking in the very human terms of patient experiences and outcomes. In other words, we’re talking about the people behind the P-values.”
