The Good, the Bad, and the Preprint

By: Max Korbmacher

Special thanks to Jay Nagaraj for editing this post.

If you want to listen along with Max as he reads through this blogpost, check out the audio file below:

What preprints are, how to understand them and why you should consider preprinting when publishing research.

Over the past decade, failed research replications and usage of questionable research practices—termed as the replication crisis—has tainted psychological research. A current initiative counteracting this crisis, is the Open Science Movement (UNESCO, 2017). This movement promotes a range of ‘good’ scientific practices such as transparency, openness, study preregistration (Nosek, Ebersole, DeHaven, & Mellor, 2018) or registered reports (Chambers, 2013, 2019; Chambers, Dienes, McIntosh, Rotshtein, & Willmes, 2015); and aims to set higher standards for making scientific claims by encouraging reproducibility in existing research (MacLean et al., 2015; Munafò et al., 2017; Vazire, 2018). Most proponents of open science—such as the Open Science Framework, the Psychological Science Accelerator, or the Student Initiative for Open Science (SIOS)—support these practices. However, one particular open science practice remains controversial: preprints. These are non-peer-reviewed manuscripts usually uploaded to designated preprint servers/online platforms. Now more than ever, the avalanche of public attention on the preprints of COVID-19 research has further polarized the reliability of preprints.

Freedom of the Preprint

Preprints allow authors to release reports of their brand-new discoveries to be read and cited by anyone, without waiting for the peer-review process to be completed (which usually takes several months). This way of bypassing the formal publication process and sharing important information directly could be instrumental, especially for urgently needed medical interventions, like the development of medical interventions for COVID-19 (Johansson, Reich, Meyers, & Lipsitch, 2018). Preprints also benefit the wider field by attracting support from journals or grant agencies (Bourne, Polka, Vale, & Kiley, 2017; Kaiser, 2017).

The Preprint Predicament – As embodied by COVID-19

Preprints are a regular part of the daily readings of many researchers. Without them, various COVID-19 interventions, such as vaccines could likely not have been developed in the record speed observed in 2020 (Thorp, 2020).

On the other hand, the COVID-19 publication rush, as marked by an all-time high in the number of manuscripts uploaded to preprint servers, had some bad eggs in the basket. This likely caused damage on different levels during the pandemic. For example, one preprint claimed that COVID-19 is not natural and hence produced artificially. This paper could even be the origin of the popular conspiracy theory spreading the same message. Other pre-printed studies simply had severe methodological flaws, but nevertheless received a lot of attention in public and political spheres.

Miscommunicating Preprints: It’s all in the fine(pre)print

Treating preprints just like their peer-reviewed counterparts comes with numerous caveats. Peer-review ensures that scientists with an expertise relevant to the paper will critically evaluate the work and its suitability for publication. Usually, manuscripts sent out for peer review will be commented on and changed accordingly several times before publication. In non-academic settings, when communicating research via press releases and news stories, unreviewed poor-quality preprints happen to be valued just as much as peer-reviewed high-quality research findings (Gunnarsdóttir, 2005). This raises concerns about damaging the reputation of science in the public eye as decision makers might be misled by unvetted research. Thus, equating preprints to peer-reviewed publications has negative consequences.

 A recent example of the COVID-19 pandemic research (Vlasschaert, Topf, & Hiremath, 2020) reveals that this “2020 line of research” on COVID-19 related topics has led to a record number in published preprints and record speed reviews. We can expect two resulting problems 1) rushed preprints flooding the internet, making it difficult for reviewers to keep up with vetting them all; and 2) many speed-reviewed manuscripts being published, leading to peer-reviewed articles that do not fulfil expectations about their quality.

Preprints in the Scientific Publication Context: To preprint or not to peer-review?

To put preprints into the scientific publication context, it is important to ask: ‘does peer-review protect from misunderstanding research?’. The answer is no. Whether reviewed or not, individuals must think critically when assessing a paper’s claims. To elaborate, we consider a great example, again from the COVID-19 sphere: the interpretation of the DANMASK-19 study by social media (Abbasi, 2020). Here, scientists studied the effectiveness of mask-wearing to prevent SARS-CoV-2 infection and found an absence of evidence for masks having a large protective effect for the wearer (Bundgaard et al., 2020). On social media however, this result has been misinterpreted and promoted as evidence for a general uselessness of wearing masks.

Peer-review doesn’t necessarily protect against poor or even fraudulent science. This can have various reasons like reviewers’ negligence or authors’ intentional use of questionable research practices. The peer-review process has also repeatedly been labelled as unfair, inefficient (Schwartz & Zamboanga, 2009) or even broken, mainly because of problematic publication incentives (Abritis & McCook, 2017; e.g., Hauser & Fehr, 2007). For instance, non-English speakers or authors with non-English sounding names, and early career researchers have been repeatedly reported to be disadvantaged (Nicholas et al., 2017; Romero-Olivares, 2019).

Publications are the currency of science; they open doors. Without them, academic careers wouldn’t advance, and both personal and project economies would take a hit. Publishers want the most exciting breakthrough papers and therefore, favour positive and novel findings over others. This leads to a biased literature (also called the file drawer problem) that undermines null results and incentivises the use of questionable research practices. This is the main reason how psychology and other fields dove headfirst into a replication crisis.

The process of peer reviews is under continuous scrutiny with room for various improvements (e.g., Fox & Lash, 2017; Schwartz & Zamboanga, 2009). Peer-reviewed literature, while promising to prevent some faults of preprints, is not yet fool proof. There is no perfect publishing format, and no scientific claim can be taken at face value. Ultimately, there is no way around critical and independent thinking when assessing scientific literature—peer-reviewed or otherwise.

The Good, the Bad, and the Preprint

Many fear that preprints offer additional opportunities to cheat the system by abusing publishing protocols and ethics (da Silva, 2018). However, the long list of retractions of peer-reviewed papers within the same year of publication shows that problems with fast research (like during the COVID-19 pandemic) cannot be singlehandedly explained by preprints (Retraction Watch, 2020). The impact of a preprint is highly dependent on its content, the publishing medium and its promoters.

Especially more technical preprints are shared mainly with experts and thereby offer more advantages (like quick information sharing and promotion of work) than disadvantages. Drawing from this example, we can argue that researchers who publish preprints should be held responsible to make it clear to the interested parties that their preprint hasn’t been peer-reviewed. Some researchers clearly state that they want to be consulted before their preprint is being covered in the media or cited. Encouraging such a practice could avoid misrepresentations and improve science communication.

Additionally, pre-printing gives scope for authors to get feedback, either directly via comments on the document uploaded to the preprint server, or indirectly via other channels such as social media or email. Some consider this kind of informal peer review process on preprint servers as ‘a more reliable peer review’ because all information is accessible and not limited to a handpicked reviewer pool. This statement is however to be taken with a pinch of salt. A preprint review can range from a single comment to a detailed in-depth review. So, the overall quantity and quality of reviews on preprint servers can be both higher and lower than a regular peer-review. Unfortunately, this depends strongly on the preprint’s visibility and interest in it. Inexperienced authors with less visible preprints will likely never receive any feedback on their preprints. Authors who know how to promote their work, who have an established audience (e.g., due to their seniority), or publish on controversial topics which cause turmoil in and around the ivory tower might have an advantage in making their preprints prominent.

Does that mean that some benefit more from preprints than others? Probably yes. This is also the case with regular peer-reviewed publications – only the rules for success differ. The future is already promising to shake up such rules: The prestigious journal eLife has recently announced to only accept manuscripts which have been preprinted previously by July 2021. Additionally, around two-thirds of preprints uploaded to bioRxiv before 2017 went on to be formally published (Abdill & Blekhman, 2019). So, maybe preprints are going to be a standard practice in the scientific publishing of the future.

Conclusion

Preprints offer a reflection of the first version of a manuscript – just as the author(s) intended the article to be, before reviewers and editors got to have a say. Comparing this ‘first publication’ with the final peer-reviewed publication helps to feel “research’s pulse of time”, giving insight into the value added by the peer-review process. Preprints should never be expected to offer the same value as peer-reviewed articles and readers should always bank on their own critical thinking when using them. Overall, preprints are freely available and invite expert discussions when topics are still hot. They remove procedural barriers from sharing knowledge. Preprints generally benefit science, thereby making it crucial for researchers to consider this medium a serious part of publishing.

References

• Abbasi, K. (2020). The curious case of the Danish mask study. BMJ, 371, m4586. doi: 10.1136/bmj.m4586
• Abdill, R. J., & Blekhman, R. (2019). Tracking the popularity and outcomes of all biorxiv preprints. ELife, 8. doi: 10.7554/eLife.45133
• Abritis, A., & McCook, A. (2017, August 11). Cash incentives for papers go global. Science, Vol. 357, p. 541. doi: 10.1126/science.357.6351.541
• Bourne, P. E., Polka, J. K., Vale, R. D., & Kiley, R. (2017). Ten simple rules to consider regarding preprint submission. PLOS Computational Biology, 13(5), e1005473. doi: 10.1371/journal.pcbi.1005473
• Bundgaard, H., Bundgaard, J. S., Raaschou-Pedersen, D. E. T., von Buchwald, C., Todsen, T., Norsk, J. B., … Iversen, K. (2020). Effectiveness of Adding a Mask Recommendation to Other Public Health Measures to Prevent SARS-CoV-2 Infection in Danish Mask Wearers. Annals of Internal Medicine. doi: 10.7326/m20-6817
• Chambers, C. D. (2013). Registered Reports: A new publishing initiative at Cortex. Cortex, 49(3), 609–610. doi: 10.1016/j.cortex.2012.12.016
• Chambers, C. D. (2019). What’s next for Registered Reports? Nature, 573(7773), 187–189. doi: 10.1038/d41586-019-02674-6
• Chambers, C. D., Dienes, Z., McIntosh, R. D., Rotshtein, P., & Willmes, K. (2015). Registered Reports: Realigning incentives in scientific publishing. Cortex, 66, 1–2. doi: 10.1016/j.cortex.2015.03.022
• da Silva, J. A. T. (2018, April 1). The preprint debate: What are the issues? Medical Journal Armed Forces India, Vol. 74, pp. 162–164. doi: 10.1016/j.mjafi.2017.08.002
• Fox, M. P., & Lash, T. L. (2017). On the Need for Quantitative Bias Analysis in the Peer-Review Process. American Journal of Epidemiology, 185(10), 865–868. doi: 10.1093/aje/kwx057
• Gunnarsdóttir, K. (2005). Scientific Journal Publications. Social Studies of Science, 35(4), 549–579. doi: 10.1177/0306312705052358
• Hauser, M., & Fehr, E. (2007). An Incentive Solution to the Peer Review Problem. PLoS Biology, 5(4), e107. doi: 10.1371/journal.pbio.0050107
• Johansson, M. A., Reich, N. G., Meyers, L. A., & Lipsitch, M. (2018). Preprints: An underutilized mechanism to accelerate outbreak science. PLOS Medicine, 15(4), e1002549. doi: 10.1371/journal.pmed.1002549
• Kaiser, J. (2017). The preprint dilemma. Science, 357(6358), 1344–1349. doi: 10.1126/science.357.6358.1344
• MacLean, D., Aleksic, J., Alexa, A., Attwood, T. K., Chue Hong, N., Dahlö, M., … Vieira, B. M. (2015). An open science peer review Oath. F1000Research, 3. doi: 10.12688/f1000research.5686.2
• Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie Du Sert, N., … Ioannidis, J. P. A. (2017, January 10). A manifesto for reproducible science. Nature Human Behaviour, Vol. 1. doi: 10.1038/s41562-016-0021
• Nicholas, D., Rodríguez-Bravo, B., Watkinson, A., Boukacem-Zeghmouri, C., Herman, E., Xu, J., … Świgoń, M. (2017). Early career researchers and their publishing and authorship practices. Learned Publishing, 30(3), 205–217. doi: 10.1002/leap.1102
• Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences of the United States of America. doi: 10.1073/pnas.1708274114
• Retraction Watch. (2020). Retracted coronavirus (COVID-19) papers. Retrieved from https://retractionwatch.com/retracted-coronavirus-covid-19-papers/
• Romero-Olivares, A. L. (2019). Reviewers, don’t be rude to nonnative English speakers. Science. doi: 10.1126/science.caredit.aaz7179
• Schwartz, S. J., & Zamboanga, B. L. (2009). The Peer-Review and Editorial System: Ways to Fix Something That Might Be Broken. Perspectives on Psychological Science, 4(1), 54–61. doi: 10.1111/j.1745-6924.2009.01106.x
• Thorp, H. H. (2020). A breakthrough for us all. Science, 370(6523), 1381–1381. doi: 10.1126/science.abg1254
• UNESCO. (2017). Open Science Movement. Retrieved August 2, 2020, from Global Open Access Portal website: http://www.unesco.org/new/en/communication-and-information/portals-and-platforms/goap/open-science-movement/
• Vazire, S. (2018). Implications of the Credibility Revolution for Productivity, Creativity, and Progress. Perspectives on Psychological Science, 13(4), 411–417. doi: 10.1177/1745691617751884
• Vlasschaert, C., Topf, J. M., & Hiremath, S. (2020, August 6). Proliferation of Papers and Preprints During the Coronavirus Disease 2019 Pandemic: Progress or Problems With Peer Review? Advances in Chronic Kidney Disease. doi: 10.1053/j.ackd.2020.08.003

Advertisement