More than one-third of the highest impact scientific journals do not offer to publish outsiders’ critiques of the papers they publish, a study has found.
The practice runs counter to recommendations from the Committee on Publication Ethics (COPE), to which most of those journals belong, and to calls from scholars for journals to be transparent and responsive when their papers are questioned. “Scientific knowledge is like a living organism that needs the nutrients of critique to survive and thrive,” says Tom Hardwicke, a metaresearcher who is moving this month to the University of Melbourne and is a co-author of the study, published on 24 August in Royal Society Open Science.
The study, one of the largest of its kind, identified 330 top-ranked journals by compiling the 15 titles with the highest journal impact factors in each of 22 scientific disciplines. It analyzed their policies as of late 2019 to early 2020, finding that 123 don’t offer formats for critiquing papers after publication, such as letters, commentaries, and online comments. Those journals included some widely cited titles, such as the Journal of the American Chemical Society (JACS), the Proceedings of the IEEE, and Remote Sensing of Environment.
Policies vary by field, the authors found. All 15 journals in clinical medicine welcomed critiques, which may reflect a recognition that flawed papers can pose direct risk to patient health. Only two math journals did.
Of the 207 journals that do publish critiques, the authors found many limit length and require submission within a few months after the paper’s publication—too strict a deadline, the authors said, because “important critiques may arise at any time.” Together, those policies raise barriers to critiquing flawed papers, the study’s authors argue. Just 2% of 2066 papers randomly selected from these journals mentioned the existence of an associated postpublication review, the team found. Hardwicke says that, based on his experience, the actual percentage of papers that could benefit from postpublication critique is far higher.
The prevalence of critique-free journals is “a little surprising,” especially because top journals often publish papers “that really push the boundaries” of science in ways that can attract criticism, says Daniel Kulp, chair of COPE, which offers guidelines to promote responsible publishing practices.
Kulp is also senior director of editorial development at the American Chemical Society (ACS), which publishes JACS and four other top journals, also COPE members, that the study found didn’t invite critiques. Kulp says he wasn’t aware of that. And ACS notes that JACS introduced a Comments section in early August, after the study was completed, that includes critiques. The other ACS journals have previously published letters and perspectives. But Hardwicke said his team didn’t count sections like these as inviting critiques unless the journal’s website or the contents of letters explicitly indicated they did.
COPE says its members “must allow debate post-publication.” So, why are some journals ignoring that call?
Hardwicke’s team did not interview journal editors, but he suggests they might feel they don’t have time to manage postpublication critiques. Editors may also believe critiques in fields such as math are too difficult to summarize in the confined space of a letter to the editor, Kulp suggests.
Another possible explanation cited by Hardwicke’s team: COPE’s guidelines allow journals to fulfill the requirement for postpublication critiques if the comments are posted on PubPeer, an independent, moderated website on which scientists anonymously dissect papers. However, some journals may not add links from the criticized paper to the comment on PubPeer—so readers of the papers might not know the critique exists.
Journals also receive critiques in another form, as requests for corrections or retractions, and the study didn’t count these as postpublication critiques. Those communications to editors are usually private, and the editors may take months or years to decide whether and how to respond. Even then, not all retracted papers are clearly marked as such.
Science, one of the journals studied, limits letters to 300 words but also provides a different format, Technical Comments, that can run up to 1000 words. In general, comments should “critique the methodology or core conclusions of the paper,” Editor-in-Chief Holden Thorp wrote in an email. The length limit reflects the view that “responses should not be the continuation of research that belongs in its own fully developed, new paper,” Thorp wrote.
The Royal Society paper offers 10 suggestions for how journals should roll out a warmer welcome for critiques. They include removing “strict” time limits, clearly telling readers how to submit critiques, and publishing them initially online to ensure speedy dissemination.
Even when critiques get published, they can face another kind of restriction—a cool reception by the paper’s authors. The study analyzed how authors responded to 58 comments on papers that had attracted at least one published postpublication critique. Only two critiques led to a published correction, the study notes. In all but three cases, the authors of the targeted article “asserted that their core claims remained unchanged despite the arguments presented.”
“It was beyond the scope of our study to examine whether author replies were appropriate and justified,” Hardwicke and his co-authors add, “but prior research has suggested that they are often inadequate.”
Source: Science Mag