Cosmos
Cosmos is a quarterly science magazine. We aim to inspire curiosity in ‘The Science of Everything’ and make the world of science accessible to everyone.
By Cosmos
Science is a process of reducing uncertainty, so we need to have confidence in its internal checks and balances, writes Clare Kenyon in this first of a special series at Cosmos Weekly.
Is it time to review the review?
Academic peer review – that fundamental structure underpinning the very integrity of science itself – is in peril.
The list of retractions and editorial issues of concern, even from the most-respected peer-reviewed journals, swells daily, exposing the underlying problem of expecting peer review to act as the gatekeeper for scientific rectitude and rigour. This is a job for which it is woefully inadequate.
Academic peer review became an integral part of the scientific publishing process in the early 1970s and quickly became synonymous with trustworthiness – both of the journal and of the science itself.
Fast forward about 50 years, and papers are no longer sent through the post, with editorial assistants cutting up and compiling reviewers’ comments to send back to authors.
The sheer number of papers submitted online to journals for review every day has grown to staggering quantities. In 2017, for instance, 10,768 articles were uploaded to the journal Nature – almost 30 per day. Currently, an estimated almost 2 million papers are published annually, in some 30,000 academic journals. That’s a lot of science to get through and a lot of specialists to find for peer review.
Editors of journals cannot possibly be across the nuances of every single specialisation within their discipline, and although journals do have lists of people in the field to which they can direct specific papers, often authors are asked to suggest appropriate reviewers upon submission of their work. As they have for decades, authors are more likely to recommend reviewers who view their work favourably or align to their particular theory more than a direct opponent’s. Although there is obvious potential for conflict of interest (including instances where authors have reviewed their own articles), as the peer-review system currently stands, the dearth of appropriately capable subject-matter experts makes it a necessity.
State of play
Most journals have statements guiding their peer-review processes, which should help navigate these complications and potentially better balance the potential for conflict. However, in many cases, these focus on the overarching criteria of science. Reviewers are typically instructed to review the overall quality and suitability of the research question, approach or experimental design method and results. But they’re rarely told to begin from the assumption that the data or experiment may be entirely fraudulent or fabricated. This is true even of some of the most prestigious journals, such as the Proceedings of the National Academy of Sciences (PNAS), Science and the Journal of the American Medical Association (JAMA), although the journal Nature does allude to the idea, asking reviewers to consider “validity” and identify and report any “flaws” in the manuscript.
As a general rule, reviewing manuscripts is an unpaid job – performed by academics as part of a long-standing tradition that likely arose from necessity rather than pure altruism.
“One of the biggest issues in peer review is the lack of incentive to do a good job.
– Dr Hannah Wardill
