A few days ago I tweeted the following:
Publishing is not the end; it’s the beginning of a general discussion. This discussion can take place a lot sooner with preprints
— Alejandro Montenegro (@aemonten) November 5, 2017
I said this, because I’ve been thinking of ways to encourage discussions, to maybe stop thinking of publications as the “end”, such that once a paper is published in a journal, I (author, editor) forget about it and don’t have to worry about it anymore. This is particularly important considering that in our current system, only a handful of people get to see a paper before it’s published in a journal, and while such peer review can be relevant for finding some of the flaws and generally improving a paper, it’s silly to think that it will find all possible problems/issues.
For this, we need the “wisdom of crowds”.
I mentioned that with preprints, in which a paper is accessible to everyone (and hence, everyone can -potentially- comment on it) as soon as the authors consider it ready, such a discussion can take place a lot sooner, as the manuscript would be available immediately, without the typical delays associated with publishing in a journal. For this particular post, I will only talk about encouraging discussions after a paper has been published in a journal and the way this could be handled (for discussion on preprints, see this). By “discussions”, I mean getting authors to engage with the people commenting on their publication.
Notably, Nikolai Slavov was just talking about this a few days ago, which prompted me to put into words some of the stuff I’ve been thinking about in the last couple of days. He published a “Point of View” in eLife some time ago in which, among other things, he discussed the following idea:
(…) journals should agree to consider non-anonymous post-publication comments submitted to certain platforms within a certain timeframe after the paper has been published. This timeframe could be as short as a few weeks or a month. Journals would be obliged to publish a response from the authors to all the substantive concerns contained in the comments. Concerns that would require a response would include the following:
-crucial controls that are missing
-major inconsistencies between the data and the main conclusions
-inconsistencies with the published literature that are not discussed in the paper
-failures by readers to reproduce analyses described in the paper
-errors in mathematical proofs.
Science is crucially based on discussions and when research is shared with the community, one should expect that a certain amount of discussion around it will take place. On Twitter, I mentioned that the lack of responses by the authors might be a matter of incentives; authors currently have no incentive to reply to any comments on a published paper. How can we create such an incentive? Discussions benefit the scientific community and should be encouraged.
One way that could be proposed, is that journals basically force authors to respond to comments raising substantial concerns, as the ones mentioned by Nikolai. How could this work? One way could be the following, which is similar to the system in place at eLife. This would work for a journal that normally publishes the peer reviewers’ comments (regardless of anonymity):
a) People who have several concerns about the validity of a published paper would reach out to the authors for comment. (As you can see, in such case, the authors have no incentive to reply).
b) If the original authors ignore or disregard the correspondence, then the people raising the concerns (let’s call them the authors of a “letter”) can contact the journal, detailing the concern(s). They should provide evidence that they have tried to contact the authors to engage in a discussion. A deadline for receiving a response should be determined before the authors can reach out to the journal. After this time has passed, the authors of the letter can decide to summarize their concerns instead of immediately writing a full analysis. This should take place within a certain time frame after the original manuscript has been published (although I’m not sure what the best time window would be). Now, if the original authors do reply to the authors of the letter in their initial correspondence, and depending on their response, we could move to either point g) or i) below.
c) The editors would evaluate the letter and determine whether further evaluation is warranted, i.e. whether the letter represents a scientifically supported challenge to the manuscript. If they agree, then, if not submitted already, the editors would invite the authors to submit a full critique. The letter may be subjected to peer review, to help the editors reach a decision.
d) After the full critique is received (and peer reviewed, in cases where this is deemed necessary), the editors would reach out to the authors and ask them to address/comment on these issues, stating that failure to reply, could result in the publication of an expression of concern, or even a retraction, depending on the concerns raised. This would be “the incentive”. The authors will be given 15 days for a “first reply”, in which they would state whether they agree to the comments and are willing to revise, or if they disagree with the critique.
e) If the editors, on the other hand, decide that a response or revision is not absolutely required, and the concerns raised do not represent a challenge to the article (either because they decided that on their initial assessment or after peer review), then the journal will offer the authors to publish their letter along the article as an online comment. The editors would explain their decision and provide the comments of the reviewers (if available) and leave it to the authors if they want to publish their original or revised letter as a comment on the original article.
In this scenario, the authors of the original article will be informed about the letter, and asked if they want to reply before the letter is published as a comment (but it would not be mandatory). This avoids “silencing” any comment that may exist about the article and will allow the community to evaluate it, such that if the community feels that the comment does indeed merit a response, then the original editorial decision could be reversed. The authors would be of course, free to publish their letter on PubMed Commons, PubPeer, a preprint server, etc. at any moment.
f) If the editors agree to the concern, and the original authors agree to the full critique or parts of it, and are willing to revise the manuscript, then they are invited to submit a revised version that includes these comments. If the editors agree to the extent of the revision, then a new version of the manuscript is published, acknowledging the changes. This would use article versioning instead of a “corrigendum” or other “legacy” terms. The new version would have a new DOI (denoting the versioning, e.g. DOI XXX.1, XXX.2, etc. This is already in place in journals like F1000 Research). In this case, as the original authors have agreed to the comments on the critique, the authors of the letter would be acknowledged in the article and their letter would be published as a non-anonymous supplementary material, together with the comments of the original peer reviewers (as they are also peer reviewers). As mentioned in the beginning, this proposal is oriented towards journals that publish the peer review comments alongside the article.
g) If alternatively, a response is received, such that it opposes the critique, and the authors refuse to make any changes, then, similarly, they will be invited to submit their full response; the response will be analyzed by the editors, subjected to peer review if necessary, and published along the article if deemed scientifically valid. In this case, the full letter would also be published. This would end the discussion.
h) If the response is deemed unsatisfactory by the editors, and the remaining concerns are major, the editors can reject the response (and publish the critique anyway). They may also decide to publish the response, just for full disclosure. At this point, the editors can decide to publish an expression of concern or retract the manuscript if they consider it necessary.
i) If the authors of the letter are not satisfied by the author response and would like to continue the discussion further, they will be invited to submit an article to the journal detailing their full critique; the manuscript would be peer reviewed and if accepted, would be published in the journal as a “Discussion”. This article will be fully linked to the original article and indexed and citable on its own. If the original authors would like to respond, then the whole cycle starts anew.
The best thing about versioning is that you will always land on the latest version of the article and you will be able to not only see the changes/updates made, but you can check out the previous versions if you want. This would be on top of being able to review the reviewer’s comments.
Now, an important point for discussion -and I’d really like your input on this- is that, if faced with such a system, would authors simply choose a journal that doesn’t have such a mechanism in place, i.e. one in which they know they won’t have to engage with criticisms? This, of course, would defeat the whole purpose. One way around this, would be that this conversation took place first at the level of societies and publishers, so that many journals would embrace it. One could argue however, that it is not a matter of authors, but it is in fact journals and editors, the ones that not only don’t have an incentive to promote such discussions, but would actually prefer not to go down this route, as it would involve more work and lead to potentially problematic discussions and in some cases, author alienation. This is another aspect to discuss.
Now, as Michael Hoffmann pointed out to me, there’s also the other aspect to all this: what’s the incentive for researchers to do post-publication peer review in the first place?
You need two to tango.
PubPeer shows us that many scientists do find it relevant to comment on work related to their research, and this might be further encouraged if journals gave a strong sign that they would consider these comments. I’d say that for at least the few people that are currently doing it, and are willing to publicly discuss and criticize a manuscript published in a journal -in a civic and scientific way-, a formal mechanism should be in place, which would guarantee a debate, or at least, disseminate their concerns.
On a related note, one might argue, “why would I go out of my way to antagonize people who will very likely be reviewers of my next paper/grant?”. One could propose the authors of the letter to be able to remain anonymous to the public and the original authors, but not to the editors., i.e. a single blind review. An argument against this, is that there may be bias in either supporting or rejecting the letter based on the identity of the letter’s authors. For instance, editors might be more willing to consider a letter coming from a big lab. A system of fully anonymous review would resemble the system in place at Pubpeer.
Anyway, I just wanted to mention some of the issues I’ve been thinking about can impact the post-publication discussion of manuscripts published in journals, and author engagement. I’d love to hear your thoughts on how we can address these issues.
For further comments and discussion, follow the threads on Twitter:
I think this might be a matter of incentives. Authors have no incentive (of those used currently in academia) for revising a published paper. You suggest that journals force authors to do it; should they retract if author’s don’t engage? That’s an incentive 🙂
— Alejandro Montenegro (@aemonten) November 14, 2017
I’d love to hear your thoughts! Latest blog post, “Publishing is not the end: thoughts about discussions on published manuscripts and author engagement”. Thanks to @michaelhoffman and @slavovLab for (pre-publication) discussions. https://t.co/gzDhxSVbFT
— Alejandro Montenegro (@aemonten) November 17, 2017