Recently, I posted an idea for a mechanism journals could have in place to not only allow people to comment on published articles, but more importantly, to get authors to actually take part of these discussions and respond, and when appropriate, amend their articles.
This is because presently, authors don’t really engage and actually, have no incentive to respond; there’s a culture of “published, ergo forgotten”. Considering the importance of discussions in science, and our present technology, there’s no need for this to continue.
Based on discussions raised by my previous post, some on Twitter (thanks Tim!), I wanted to propose another mechanism (which might actually be way simpler than the other) that could be employed by journals to be able to make use of the ‘wisdom of crowds’ and encourage authors to take part of discussions raised by the community. Similarly, this mechanism makes the community be a part of the whole process and encourages participation, as their comments can certainly make a difference.
As I stated in that previous post,
This is particularly important considering that in our current system, only a handful of people get to see a paper before it’s published in a journal, and while such peer review can be relevant for finding some of the flaws and generally improving a paper, it’s silly to think that it will find all possible problems/issues.
Allowing the community to comment on a published paper and knowing that their comments can have a real impact, is the important issue. I strongly support preprints and think that it would be crucial to get feedback at that stage, but at present, that is not common. For the sake of discussion, let’s again propose a way for encouraging authors to engage with the discussions taking place around their paper published in a journal.
Here, an author would submit the article to a journal (and at the same time, to a preprint server ;)), where the editor would subject it to pre-publication peer review. The editor would choose the reviewers (s)he likes best for this manuscript, and know are suitable experts to evaluate this work. If after the necessary rounds of review (ideally, consultative peer review or one with ‘cross commenting’), the editor considers the article to be acceptable for the journal, the manuscript would be “Conditionally Accepted”, which would basically mean that the manuscript can move onto the next stage of the process: open peer review.
I know this term can mean different things to different people (like ‘epigenetics’ 😉 ), but here, I mean “open to whomever wants to participate”. In this setting, the peer-reviewed and editor-approved, non-typesetted version, is made public and the community is invited to comment on the manuscript, within a certain time window. Anyone could comment and if they prefer, they can maintain their anonymity. A system could be put in place for these comments to have a DOI, if the community would like that. People commenting are also free to cross-post their comments on other platforms, for instance, if the manuscript is deposited on a preprint server.
During that window, the authors can address individual comments if they want, or wait to address them all together at the end of the time window. The authors may decide to revise their manuscript in response to these comments or simply respond to them.
A certain amount of time (which would need to be defined) after the window closes, the authors must submit their responses to the comments posted by the community. The editor will review their responses and decide, at this point, the next step: a) accept (the manuscript would be moved into production), b) revisions, or c) reject. If the manuscript is accepted, the reviewers’ comments (anonymous if they prefer so) and author responses, would be posted along the manuscript.
This would avoid some of the problems raised on the previously proposed approach, which involves a series of steps, and is overall, more cumbersome and would likely involve a lot more work for all parties involved.
Problems with this new approach however, include: 1) the time from submission to acceptance will increase and 2) original peer reviewers might feel they wasted their time, particularly if the ‘open peer review’ stage leads to a decision that is different from their recommendation (I’d argue that happens today, anyway). This is of course, disregarding the problem of ‘authors have no time to reply to comments’, commonly raised against any proposal that attempts to encourage authors to participate in discussions of their manuscripts.
Remember, the idea is to encourage authors to respond to concerns raised by the community. In this way, the incentive is clear: no response, no publication. If there are no comments, the paper is published as approved in the pre-publication peer review process.
Yes, at the beginning, most papers would have no comments on the ‘open peer review’ stage, but as Nikolai said:
Even big things start small… even if participation rate is low at the beginning, it might grow. The @biorxivpreprint is a great example of low participation rate exploding into a mainstream cultural shift that is transforming scientific communication
— Nikolai Slavov (@slavovLab) November 17, 2017
In a way, this system is a modified version of what is currently in place at Atmospheric Chemistry and Physics (ACP),
Of course, one could also propose a system in which the “pre-publication” peer review stage is discarded altogether, as in ACP. In this scenario, the editor would still personally invite some experts to review (and again, they could do it anonymously, in this case, in a single blind fashion), just to make sure every article is reviewed by some experts. Is this something the life science community would be willing to try? The experience with F1000 Research and the like seems to suggest so.
Lastly, note that this is not something to compete with or replace post-pub peer review. That would still exist and could play a relevant role after the time window of open peer review has ended. Also, researchers would be welcome to publish rebuttals of published papers in the same journal, using mechanisms as the one described before.