Researchers spend a lot of time doing peer review, and by a lot we are talking about over 100 million hours per year (estimate for 2020 by Aczel et al.). It is a complex and time consuming process that is often presented as a pillar to science dissemination, because of its function to scrutinize research papers to check whether they contain any flaws, oversights, or they meet certain criteria for novelty or advance, before the article appears in a journal. And yet a lot of this happens in the backstage of the journal’s process, hidden from public view. This approach to peer review results in inefficiencies in the system, where the same paper can undergo multiple rounds of review by different reviewers at different journals following rejection by one journal - another estimate suggests that over 15 million hours per year are spent on these redundant reviews. Keeping the reviews hidden also makes it difficult to credit review activities, the review process may vary from one journal to another or even from one manuscript to another within the same journal, so what is it that we would be rewarding? Reviewers can claim they did a review for a journal, but beyond this statement of a report being completed, we know very little as to whether the review was useful, constructive, or it led to any changes in the manuscript.
We have seen major changes in scholarly communication in the last couple of decades: open access, the drive for open research data, adoption of preprints across disciplines and more. All of these have the common theme of opening up different elements of the research activities. Given the time and effort invested in review activities, should we not also strive to open up review?
Leaving the black box behind
Over the last few years, there has been a push for more transparency in journal peer review and many journals now publish the review reports for published articles (Wolfram et al.). This is a very welcome development, but it is not yet the norm across scholarly journals. Publishing reports with journal articles also means that we only get to see the reviews for those papers that made it to publication, and only the reviews for the submission to that particular journal - any review activity for earlier versions of the work is still lost.
In this context, peer reviews are still very much focused on informing an editorial decision. Most reviewers contribute their time to peer review out of a motivation to give back to the community, to support this scholarly activity as they expect others will do for their own manuscripts. This is a laudable motivation, but the focus on journal gatekeeping means that many authors feel disempowered in the peer review process, where they feel they should bend to every request by reviewers to get the acceptance decision they long for.
The good news is that we do not need to wait for a protracted system change, there are ways in which researchers can already make their reviews both more open and useful for the community:
- Only review for journals that publish the review reports for the articles they publish.
- When reviewing for a journal, if the paper has an associated preprint, post the review publicly on the preprint, as proposed by Ludo Waltman, James Fraser and Cooper Smout. This attaches the evaluation to the scientific work itself, independent of the eventual journal’s decision.
- Post your comments and reviews on preprints. Researchers discuss research papers in many formats and forums, independent of a journal's editorial process. They discuss the latest paper in their research area with collaborators, debate articles at journal clubs or social media, or discuss research at conferences. All that feedback is valuable, and if made public it would not only benefit the authors but also other members of the research community and the public.
I realize that not everyone may feel comfortable signing the reviews, so in all the scenarios above I refer to the publication of reports, which may or may not include the reviewer’s name. In the context of preprint reviews, there are options available to post reviews anonymously e.g. by posting under a pseudonym if the platform provides this option or by asking a colleague willing to act as an intermediate to post the review.
Reviews open by and for the community
The possibility of posting reviews on preprints places the agency back with researchers, who can choose when and how they wish to make their comments public, without relying on an editor’s decision or whether the journal has built-in workflows to publish the report. A review is the researcher’s scholarly contribution, posting the reviews publicly not only maximizes the value of the review for the broad community, but also opens up new ways of providing credit for review activities. In the preprint environment, we can judge the content of the review by ourselves, and start making new decisions about what we value in reviews and what we’d like to give recognition for.
Open review outside the journal also opens new opportunities for authors. In preprint review, there is no designated review template that asks for comments in a Major, Minor and publication recommendation format. The author of a preprint may be seeking community input on what direction to take with their research, or may be interested in collaborations to complete further experiments. By requesting feedback on the work in the format that best suits their research, the author becomes an active participant in the exchange, and not just the recipient of a thumbs up or down decision from a journal.
This brings a range of exciting new options, but also makes the environment more complex than the traditional ‘peer review’ umbrella we generally refer to in the journal context. For open and new review modalities to thrive we will need new ways to codify activities, and to designate what we value in peer review and how we will reward it. There have been some initial steps, for example, the FAST principles for preprint feedback (disclaimer, I am a co-author of the principles) provide a set of principles for the expected behaviors when creating and responding to preprint feedback. We will need many more conversations around open review: what do different stakeholders consider a ‘review’? What do we value in reviews? What does review look like for different research objects? And we should also recognize that the answers to these questions may be different for different groups.
We may not yet know what the future of review will look like, but we know it will be much more open than it is now. Let’s lead by example and start now by making our reviews open.
Copyright © 2022 Iratxe Puebla. Distributed under the terms of the Creative Commons Attribution 4.0 License.
