Discredited Research

For John, BLUF:  Just because it is in a "Scientific Journal" doesn't mean it is true or correct.

Tuesday last I posted here on an apparent problem with research being done and published, especially in the field of medicine.  As a response I received a comment off-line from Dr Tracy Perez Koehlmoos, who specializes in health systems and policy research. She previously headed the Health and Family Planning Systems Programme at ICDDR,B in Dhaka, Bangladesh, and now works for the Department of Defense, in Washington.  I decided it was long enough it deserved its own blog post.  And without using the quote indents.  So, from Dr Koehlmoos:

This month the open access journal with the highest impact factor:  PLoS Med (short for Public Library of Science Medicine) will publish a special issue on research misconduct.  The main articles are broken down into research misconduct in high-income countries and research misconduct in low- and middle-income countries (LMIC).

I am second author on the LMIC article—my co-authors are from Nigeria, China and the UK. There was a fifth coauthor who would have brought a lot to the paper, but he/she withdrew from the publication due to concerns about the potential impact on his/her career due to content in the article.

There is the assumption that published research studies are good.  That being published alone is enough to speak for the quality of the study.  Or I should say, among those who do not or rarely publish, that published research studies are good—in terms of having been critically reviewed during the peer review process.  I deal with this frequently when people talk about program design or evaluating programs.  The fall back statement seems to be—"We will look at the published literature"—as if that constitutes quality.  My follow-on point is always, "And what tool will you use to critically appraise the published literature?" as well as "And how will you identify unpublished studies of the same intervention?"  (I must be such a buzz kill to those around me, but don't the people whom I serve deserve this level of scrutiny in decision making for health and health policy related items?  Don't we all want healthcare and health policy that has gone through similar scrutiny?)

Let me tell you now—there is a LOT of garbage out there, a lot of studies that are published, even in top journals, in which the data tells us one thing and the authors tell us another in the discussion and conclusion section.  How do I know this?  I am a systematic reviewer.  Critical appraisal of included studies is a hallmark of systematic review, which is about a great deal more than just meta-analysis of randomized control trials.

When I am doing a systematic review and judge someone's published paper to be of poor quality (in terms of data collection, data analysis or the presentation of results, discussion and/or conclusion) and then I go out and write about it—and publish it in my review, it does NOT make me the favorite person of the authors of the publication.

A friend of an author whose work is confusing—the numerical analysis does not add up to what is argued in the discussion and summarized in the conclusion sections—once said to me, "He was up to his elbows in data.  You don't know how hard it was for him to get to where he could write a paper."  And?  While I empathize completely as a researcher and author, in my systematic review I can only say that the paper in question does nothing to prove the effectiveness of the intervention.

Could it be research misconduct?  It might be...if it was done intentionally. The myth that researchers do not have an agenda when it comes to publishing is wrong.  Most researchers fund their work through competitive grants—so that it is a constant fight for funding—how do you generate funds?  By publishing your work and building the reputation of your program.  The scientist going for publication then has an inherent interest in making sure that his or her work gets published in the highest quality peer review publication so that it will be seen and read and cited by others.

Health research is a big business and there are many scientists/researchers with colossal egos interested in promoting their own agenda.  This fact used to floor me, especially in global health where I have always thought we were all working together to improve the lives of the poor...I was wrong; I was naive and I definitely learned the hard way.

Perhaps more relevant to a US audience, think of the now discredited work on the links between the measles vaccine and autism.  This was research misconduct.  However, years after the physician researcher was discredited and the article retracted, there are well-intentioned but misinformed parents who refuse to immunize their children—thus putting their child and their neighbors at risk.

It is great to see that research misconduct is getting press time.  We must be constantly vigilant.

The End.

Thank you, Tracey.

Regards  —  Cliff