The effect of publishing peer review reports on referee behavior in five scholarly journals

Authors: Giangiacomo Bravo, Francisco Grimaldo, Emilia López-Iñesta, Bahar Mehmani, & Flaminio Squazzoni

Abstract: To increase transparency in science, some scholarly journals are publishing peer review reports. But it is unclear how this practice affects the peer review process. Here, we examine the effect of publishing peer review reports on referee behavior in five scholarly journals involved in a pilot study at Elsevier. By considering 9,220 submissions and 18,525 reviews from 2010 to 2017, we measured changes both before and during the pilot and found that publishing reports did not significantly compromise referees’ willingness to review, recommendations, or turn-around times. Younger and non-academic scholars were more willing to accept to review and provided more positive and objective recommendations. Male referees tended to write more constructive reports during the pilot. Only 8.1% of referees agreed to reveal their identity in the published report. These findings suggest that open peer review does not compromise the process, at least when referees are able to protect their anonymity.

Citation:   Lemke S, Mehrazar M, Mazarakis A and Peters I (2019) “When You Use Social Media You Are Not Working”: Barriers for the Use of Metrics in Social Sciences. Front. Res. Metr. Anal. 3:39. doi: 10.3389/frma.2018.00039



A multi-disciplinary perspective on emergent and future innovations in peer review

Authors: Jonathan P Tennant, Jonathan M Dugan, Daniel Graziotin, Damien C Jacques, François Waldner, Daniel Mietchen, Yehia Elkhatib, Lauren B Collister, Christina K Pikas, Tom Crick, Paola Masuzzo, Anthony Caravaggi, Devin R Berg, Kyle E Niemeyer, Tony Ross-Hellauer, Sara Mannheimer, Lillian Rigling, Daniel S Katz, Bastian Greshake Tzovaras, Josmel Pacheco-Mendoza, Nazeefa Fatima, Marta Poblet, Marios Isaakidis, Dasapta Erwin Irawan, Sébastien Renaut, Christopher R Madan, Lisa Matthias, Jesper Nørgaard Kjær, Daniel Paul O’Donnell, Cameron Neylon, Sarah Kearns, Manojkumar Selvaraju, Julien Colomb

Abstract: Peer review of research articles is a core part of our scholarly communication system. In spite of its importance, the status and purpose of peer review is often contested. What is its role in our modern digital research and communications infrastructure? Does it perform to the high standards with which it is generally regarded? Studies of peer review have shown that it is prone to bias and abuse in numerous dimensions, frequently unreliable, and can fail to detect even fraudulent research. With the advent of web technologies, we are now witnessing a phase of innovation and experimentation in our approaches to peer review. These developments prompted us to examine emerging models of peer review from a range of disciplines and venues, and to ask how they might address some of the issues with our current systems of peer review. We examine the functionality of a range of social Web platforms, and compare these with the traits underlying a viable peer review system: quality control, quantified performance metrics as engagement incentives, and certification and reputation. Ideally, any new systems will demonstrate that they out-perform and reduce the biases of existing models as much as possible. We conclude that there is considerable scope for new peer review initiatives to be developed, each with their own potential issues and advantages. We also propose a novel hybrid platform model that could, at least partially, resolve many of the socio-technical issues associated with peer review, and potentially disrupt the entire scholarly communication system. Success for any such development relies on reaching a critical threshold of research community engagement with both the process and the platform, and therefore cannot be achieved without a significant change of incentives in research environments.

Citation: Tennant JP, Dugan JM, Graziotin D et al. A multi-disciplinary perspective on emergent and future innovations in peer review [version 3; referees: 2 approved]. F1000Research 2017, 6:1151


Source: F1000 Research

Scholarship as an Open Conversation: Utilizing Open Peer Review in Information Literacy Instruction

Authors: Emily Ford

Abstract: This article explores the ACRL Framework for Information Literacy’s frame, Scholarship as a Conversation. This frame asserts that information literate students have the disposition, skills, and knowledge to recognize and participate in disciplinary scholarly conversations. By investigating the peer-review process as part of scholarly conversations, this article provides a brief literature review on peer review in information literacy instruction, and argues that by using open peer review (OPR) models for teaching, library workers can allow students to gain a deeper understanding of scholarly conversations. OPR affords students the ability to begin dismantling the systemic oppression that blinded peer review and the traditional scholarly publishing system reinforce. Finally, the article offers an example classroom activity using OPR to help students enter scholarly conversations, and recognize power and oppression in scholarly publishing.

Citation: Ford, E. (2018).Scholarship as an Open Conversation: Utilizing Open Peer Review in Information Literacy Instruction. In the Library with the Lead Pipe.


Source: In the Library with the Lead Pipe

“Let the community decide”? The vision and reality of soundness-only peer review in open-access mega-journals

Authors: Valerie Spezi, Simon Wakeling, Stephen Pinfield, Jenny Fry, Claire Creaser, Peter Willett

The purpose of this paper is to better understand the theory and practice of peer review in open-access mega-journals (OAMJs). OAMJs typically operate a “soundness-only” review policy aiming to evaluate only the rigour of an article, not the novelty or significance of the research or its relevance to a particular community, with these elements being left for “the community to decide” post-publication.

The paper reports the results of interviews with 31 senior publishers and editors representing 16 different organisations, including 10 that publish an OAMJ. Thematic analysis was carried out on the data and an analytical model developed to explicate their significance.

Findings suggest that in reality criteria beyond technical or scientific soundness can and do influence editorial decisions. Deviations from the original OAMJ model are both publisher supported (in the form of requirements for an article to be “worthy” of publication) and practice driven (in the form of some reviewers and editors applying traditional peer review criteria to OAMJ submissions). Also publishers believe post-publication evaluation of novelty, significance and relevance remains problematic.

The study is based on unprecedented access to senior publishers and editors, allowing insight into their strategic and operational priorities. The paper is the first to report in-depth qualitative data relating specifically to soundness-only peer review for OAMJs, shedding new light on the OAMJ phenomenon and helping inform discussion on its future role in scholarly communication. The paper proposes a new model for understanding the OAMJ approach to quality assurance, and how it is different from traditional peer review.

Citation: Valerie Spezi, Simon Wakeling, Stephen Pinfield, Jenny Fry, Claire Creaser, Peter Willett (2018). ““Let the community decide”? The vision and reality of soundness-only peer review in open-access mega-journals.” Journal of Documentation, Vol. 74 Issue: 1, pp.137-161.


Source: Journal of Documentation

A prospective study on an innovative online forum for peer reviewing of surgical science

Authors:  Almquist M, von Allmen RS, Carradice D, Oosterling SJ, McFarlane K, Wijnhoven B

Peer review is important to the scientific process. However, the present system has been criticised and accused of bias, lack of transparency, failure to detect significant breakthrough and error. At the British Journal of Surgery (BJS), after surveying authors’ and reviewers’ opinions on peer review, we piloted an open online forum with the aim of improving the peer review process.

In December 2014, a web-based survey assessing attitudes towards open online review was sent to reviewers with a BJS account in Scholar One. From April to June 2015, authors were invited to allow their manuscripts to undergo online peer review in addition to the standard peer review process. The quality of each review was evaluated by editors and editorial assistants using a validated instrument based on a Likert scale.

The survey was sent to 6635 reviewers. In all, 1454 (21.9%) responded. Support for online peer review was strong, with only 10% stating that they would not subject their manuscripts to online peer review. The most prevalent concern was about intellectual property, being highlighted in 118 of 284 comments (41.5%). Out of 265 eligible manuscripts, 110 were included in the online peer review trial. Around 7000 potential reviewers were invited to review each manuscript. In all, 44 of 110 manuscripts (40%) received 100 reviews from 59 reviewers, alongside 115 conventional reviews. The quality of the open forum reviews was lower than for conventional reviews (2.13 (± 0.75) versus 2.84 (± 0.71), P<0.001).

Open online peer review is feasible in this setting, but it attracts few reviews, of lower quality than conventional peer reviews.

Citation: Almquist M, von Allmen RS, Carradice D, Oosterling SJ, McFarlane K, Wijnhoven B (2017) A prospective study on an innovative online forum for peer reviewing of surgical science. PLoS ONE 12(6): e0179031.


Looking into Pandora’s Box: The Content of Sci-Hub and its Usage

Author: Bastian Greshake

Abstract: Despite the growth of Open Access, potentially illegally circumventing paywalls to access scholarly publications is becoming a more mainstream phenomenon. The web service Sci-Hub is amongst the biggest facilitators of this, offering free access to around 62 million publications. So far it is not well studied how and why its users are accessing publications through Sci-Hub. By utilizing the recently released corpus of Sci-Hub and comparing it to the data of  ~28 million downloads done through the service, this study tries to address some of these questions. The comparative analysis shows that both the usage and complete corpus is largely made up of recently published articles, with users disproportionately favoring newer articles and 35% of downloaded articles being published after 2013. These results hint that embargo periods before publications become Open Access are frequently circumnavigated using Guerilla Open Access approaches like Sci-Hub. On a journal level, the downloads show a bias towards some scholarly disciplines, especially Chemistry, suggesting increased barriers to access for these. Comparing the use and corpus on a publisher level, it becomes clear that only 11% of publishers are highly requested in comparison to the baseline frequency, while 45% of all publishers are significantly less accessed than expected. Despite this, the oligopoly of publishers is even more remarkable on the level of content consumption, with 80% of all downloads being published through only 9 publishers. All of this suggests that Sci-Hub is used by different populations and for a number of different reasons, and that there is still a lack of access to the published scientific record. A further analysis of these openly available data resources will undoubtedly be valuable for the investigation of academic publishing.

Citation:  Greshake B.Looking into Pandora’s Box: The Content of Sci-Hub and its Usage.” F1000Research 2017, 6:541. (doi: 10.12688/f1000research.11366.1) .