Part 2 of “Publishing is not the end: thoughts about discussions on published manuscripts and author engagement”

Recently, I posted an idea for a mechanism journals could have in place to not only allow people to comment on published articles, but more importantly, to get authors to actually take part of these discussions and respond, and when appropriate, amend their articles.

This is because presently, authors don’t really engage and actually, have no incentive to respond; there’s a culture of “published, ergo forgotten”. Considering the importance of discussions in science, and our present technology, there’s no need for this to continue.

Based on discussions raised by my previous post, some on Twitter (thanks Tim!), I wanted to propose another mechanism (which might actually be way simpler than the other) that could be employed by journals to be able to make use of the ‘wisdom of crowds’ and encourage authors to take part of discussions raised by the community. Similarly, this mechanism makes the community be a part of the whole process and encourages participation, as their comments can certainly make a difference.

As I stated in that previous post,

This is particularly important considering that in our current system, only a handful of people get to see a paper before it’s published in a journal, and while such peer review can be relevant for finding some of the flaws and generally improving a paper, it’s silly to think that it will find all possible problems/issues.

Allowing the community to comment on a published paper and knowing that their comments can have a real impact, is the important issue. I strongly support preprints and think that it would be crucial to get feedback at that stage, but at present, that is not common. For the sake of discussion, let’s again propose a way for encouraging authors to engage with the discussions taking place around their paper published in a journal.

Here, an author would submit the article to a journal (and at the same time, to a preprint server ;)), where the editor would subject it to pre-publication peer review. The editor would choose the reviewers (s)he likes best for this manuscript, and know are suitable experts to evaluate this work. If after the necessary rounds of review (ideally, consultative peer review or one with ‘cross commenting’), the editor considers the article to be acceptable for the journal, the manuscript would be “Conditionally Accepted”, which would basically mean that the manuscript can move onto the next stage of the process: open peer review.

I know this term can mean different things to different people (like ‘epigenetics’ 😉  ), but here, I mean “open to whomever wants to participate”. In this setting, the peer-reviewed and editor-approved, non-typesetted version, is made public and the community is invited to comment on the manuscript, within a certain time window. Anyone could comment and if they prefer, they can maintain their anonymity. A system could be put in place for these comments to have a DOI, if the community would like that. People commenting are also free to cross-post their comments on other platforms, for instance, if the manuscript is deposited on a preprint server.

During that window, the authors can address individual comments if they want, or wait to address them all together at the end of the time window. The authors may decide to revise their manuscript in response to these comments or simply respond to them.

A certain amount of time (which would need to be defined) after the window closes, the authors must submit their responses to the comments posted by the community. The editor will review their responses and decide, at this point, the next step: a) accept (the manuscript would be moved into production), b) revisions, or c) reject. If the manuscript is accepted, the reviewers’ comments (anonymous if they prefer so) and author responses, would be posted along the manuscript.

This would avoid some of the problems raised on the previously proposed approach, which involves a series of steps, and is overall, more cumbersome and would likely involve a lot more work for all parties involved.

Problems with this new approach however, include: 1) the time from submission to acceptance will increase and 2) original peer reviewers might feel they wasted their time, particularly if the ‘open peer review’ stage leads to a decision that is different from their recommendation (I’d argue that happens today, anyway). This is of course, disregarding the problem of ‘authors have no time to reply to comments’, commonly raised against any proposal that attempts to encourage authors to participate in discussions of their manuscripts.

Remember, the idea is to encourage authors to respond to concerns raised by the community. In this way, the incentive is clear: no response, no publication. If there are no comments, the paper is published as approved in the pre-publication peer review process.

Yes, at the beginning, most papers would have no comments on the ‘open peer review’ stage, but as Nikolai said:

In a way, this system is a modified version of what is currently in place at Atmospheric Chemistry and Physics (ACP),

 

Of course, one could also propose a system in which the “pre-publication” peer review stage is discarded altogether, as in ACP. In this scenario, the editor would still personally invite some experts to review (and again, they could do it anonymously, in this case, in a single blind fashion), just to make sure every article is reviewed by some experts. Is this something the life science community would be willing to try? The experience with F1000 Research and the life seems to suggest so.

Lastly, note that this is not something to compete  with or replace post-pub peer review. That would still exist and could play a relevant role after the time window of open peer review has ended. Also, researchers would be welcome to publish rebuttals of published papers in the same journal, using mechanisms as the one described before.

Thoughts?

Advertisements
  1. #1 by Manu Saunders on November 19, 2017 - 04:53

    Interesting approach. I think the main challenge would be motivating people to comment in the open stage. For example, PLOS, PeerJ and preprint servers already use this comment approach for published articles and there appears to be very little uptake of this feature. Also, I think the same issue with preprints applies here too – there are potential scicomm issues arising from making possibly flawed research public before it is peer reviewed (although, here there is the second stage of standard peer review, which may alleviate this risk somewhat).

    • #2 by Alejandro Montenegro-Montero on November 19, 2017 - 11:04

      Thanks for commenting! I agree that the main challenge is to get people to participate. That’s what I quoted Nikolai in the end of the post. In any case, PubPeer shows us that there are people interested in commenting on papers related to their areas of research.

      What is missing, in a way, is an incentive to participate. I think that in this approach, there might be an extra motivation to participate, as here, the comments can actually make a difference, i.e. can have a direct impact on the published manuscript. This differs from the post-publication commenting system currently in place in other journals.

      Regarding your last part, certainly! I understand that point. The paper would be made public only after it has gone regular peer review. If after the necessary rounds of review, the editor thinks it is publishable, it would only then be made publicly available. This might partially mitigate that concern.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: