A study finds that papers describing the results of clinical trials often fail to properly report outcomes.
ISTOCK.COM/PEOPLEIMAGES
By Jocelyn Kaiser
It’s a well-known problem with clinical trials: Researchers start out saying they will look for a particular outcome—heart attacks, for example—but then report something else when they publish their results. That practice can make a drug or treatment look like it’s safer or more effective than it actually is. Now, a systematic effort to find out whether major journals are complying with their own pledge to ensure that outcomes are reported correctly has found many are falling down on the job—and both journals and authors are full of excuses.
When journals and researchers were asked to correct studies, the responses “were fascinating, and alarming. Editors and researchers routinely misunderstand what correct trial reporting looks like,” says project leader Ben Goldacre, an author and physician at the University of Oxford in the United Kingdom and a proponent of transparency in drug research.
Starting 4 years ago, his team’s Centre for Evidence-Based Medicine Outcome Monitoring Project (COMPare) project examined all trials published over 6 weeks in five journals: Annals of Internal Medicine, The BMJ, JAMA, The Lancet, and The New England Journal of Medicine (NEJM). The study topics ranged from the health effects of drinking alcohol for diabetics to a comparison of two kidney cancer drugs. All five journals have endorsed long-established Consolidated Standards of Reporting Trials (CONSORT) guidelines. One CONSORT rule is that authors should describe the outcomes they plan to study before a trial starts and stick to that list when they publish the trial.
But only nine of 67 trials published in the five journals reported outcomes correctly, the COMPare team reported on 14 February in the journal Trials. One-fourth didn’t correctly report the primary outcome they set out to measure and 45% didn’t properly report all secondary outcomes; others added new outcomes. (This varied by journal: only 44% of trials in Annals correctly reported the primary outcome, compared with 96% of NEJM trials.)
When the COMPare team wrote the journals about the problematic papers, only 23 of the 58 letters were published. Annals and The BMJ published all of them, The Lancet accepted 80%, and NEJM and JAMA rejected them all. NEJM editors explained that their editors and peer reviewers decide which outcomes will be reported. While some of the CONSORT rules are “useful,” they wrote, authors aren’t required to comply. Other editors didn’t seem to understand that trial researchers can switch outcomes if they disclose the change. JAMA and NEJM said they didn’t always have space to publish all outcomes.
When trial authors responded to the letters that did make it into print, their comments were full of “inaccurate or problematic statements and misunderstandings,” the COMPare team found in a companion paper in Trials. Like editors, many authors misunderstood the CONSORT rules, as well as the role of public registries for sharing a trial’s plan. Some attacked the COMPare project as “outside the research community.” Others brushed off the criticisms, grumbling about how difficult their work was. Still others denied that they left out any outcomes, the authors state.
The COMPare team writes that it hopes journals will be inspired to better enforce CONSORT and revisit their standards for publishing letters. “We hope that editors will respond positively, constructively and thoughtfully to our findings,” Goldacre says.
Source: Science Mag