The F3-index. Valuing reviewers for scholarly journals

Authors: Federico Bianchi, Francisco Grimaldo, Flaminio Squazzoni

Abstract: This paper presents an index that measures reviewer contribution to editorial processes of scholarly journals. Following a metaphor of ranking algorithms in sports tournaments, we created an index that considers reviewers on different context-specific dimensions, i.e., report delivery time, the length of the report and the alignment of recommendations to editorial decisions. To test the index, we used a dataset of peer review in a multi-disciplinary journal, including 544 reviewers on 606 submissions in six years. Although limited by sample size, the test showed that the index identifies outstanding contributors and weak performing reviewers efficiently. Our index is flexible, contemplates extensions and could be incorporated into available scholarly journal management tools. It can assist editors in rewarding high performing reviewers and managing editorial turnover.

Citation:   Lemke S, Mehrazar M, Mazarakis A and Peters I (2019) “When You Use Social Media You Are Not Working”: Barriers for the Use of Metrics in Social Sciences. Front. Res. Metr. Anal. 3:39. doi: 10.3389/frma.2018.00039

View

 

s

Source: Journal of Informetrics

The effect of publishing peer review reports on referee behavior in five scholarly journals

Authors: Giangiacomo Bravo, Francisco Grimaldo, Emilia López-Iñesta, Bahar Mehmani, & Flaminio Squazzoni

Abstract: To increase transparency in science, some scholarly journals are publishing peer review reports. But it is unclear how this practice affects the peer review process. Here, we examine the effect of publishing peer review reports on referee behavior in five scholarly journals involved in a pilot study at Elsevier. By considering 9,220 submissions and 18,525 reviews from 2010 to 2017, we measured changes both before and during the pilot and found that publishing reports did not significantly compromise referees’ willingness to review, recommendations, or turn-around times. Younger and non-academic scholars were more willing to accept to review and provided more positive and objective recommendations. Male referees tended to write more constructive reports during the pilot. Only 8.1% of referees agreed to reveal their identity in the published report. These findings suggest that open peer review does not compromise the process, at least when referees are able to protect their anonymity.

Citation:   Lemke S, Mehrazar M, Mazarakis A and Peters I (2019) “When You Use Social Media You Are Not Working”: Barriers for the Use of Metrics in Social Sciences. Front. Res. Metr. Anal. 3:39. doi: 10.3389/frma.2018.00039


View

 

Gender and international diversity improves equity in peer review


Authors: Dakota Murray, Kyle Siler, Vincent Lariviére, Wei Mun Chan, Andrew M. Collings, Jennifer Raymond, Cassidy R Sugimoto

Abstract: The robustness of scholarly peer review has been challenged by evidence of disparities in publication outcomes based on author’s gender and nationality. To address this, we examine the peer review outcomes of 23,873 initial submissions and 7,192 full submissions that were submitted to the biosciences journal eLife between 2012 and 2017. Women and authors from nations outside of North America and Europe were underrepresented both as gatekeepers (editors and peer reviewers) and last authors. We found a homophilic interaction between the demographics of the gatekeepers and authors in determining the outcome of peer review; that is, gatekeepers favor manuscripts from authors of the same gender and from the same country. The acceptance rate for manuscripts with male last authors was significantly higher than for female last authors, and this gender inequity was greatest when the team of reviewers was all male; mixed-gender gatekeeper teams lead to more equitable peer review outcomes. Similarly, manuscripts were more likely to be accepted when reviewed by at least one gatekeeper with the same national affiliation as the corresponding author. Our results indicated that homogeneity between author and gatekeeper gender and nationality is associated with the outcomes of scientific peer review. We conclude with a discussion of mechanisms that could contribute to this effect, directions for future research, and policy implications. Code and anonymized data have been made available at https://github.com/murrayds/elife-analysis

Citation: Dakota Murray, Kyle Siler, Vincent Lariviére, Wei Mun Chan, Andrew M. Collings, Jennifer Raymond, Cassidy R. Sugimoto (2018). Gender and international diversity improves equity in peer review.

View

Source: bioRxiv

Scholarship as an Open Conversation: Utilizing Open Peer Review in Information Literacy Instruction


Authors: Emily Ford

Abstract: This article explores the ACRL Framework for Information Literacy’s frame, Scholarship as a Conversation. This frame asserts that information literate students have the disposition, skills, and knowledge to recognize and participate in disciplinary scholarly conversations. By investigating the peer-review process as part of scholarly conversations, this article provides a brief literature review on peer review in information literacy instruction, and argues that by using open peer review (OPR) models for teaching, library workers can allow students to gain a deeper understanding of scholarly conversations. OPR affords students the ability to begin dismantling the systemic oppression that blinded peer review and the traditional scholarly publishing system reinforce. Finally, the article offers an example classroom activity using OPR to help students enter scholarly conversations, and recognize power and oppression in scholarly publishing.

Citation: Ford, E. (2018).Scholarship as an Open Conversation: Utilizing Open Peer Review in Information Literacy Instruction. In the Library with the Lead Pipe.

View

Source: In the Library with the Lead Pipe

A prospective study on an innovative online forum for peer reviewing of surgical science


Authors:  Almquist M, von Allmen RS, Carradice D, Oosterling SJ, McFarlane K, Wijnhoven B

Abstract:
Background
Peer review is important to the scientific process. However, the present system has been criticised and accused of bias, lack of transparency, failure to detect significant breakthrough and error. At the British Journal of Surgery (BJS), after surveying authors’ and reviewers’ opinions on peer review, we piloted an open online forum with the aim of improving the peer review process.

Methods
In December 2014, a web-based survey assessing attitudes towards open online review was sent to reviewers with a BJS account in Scholar One. From April to June 2015, authors were invited to allow their manuscripts to undergo online peer review in addition to the standard peer review process. The quality of each review was evaluated by editors and editorial assistants using a validated instrument based on a Likert scale.

Results
The survey was sent to 6635 reviewers. In all, 1454 (21.9%) responded. Support for online peer review was strong, with only 10% stating that they would not subject their manuscripts to online peer review. The most prevalent concern was about intellectual property, being highlighted in 118 of 284 comments (41.5%). Out of 265 eligible manuscripts, 110 were included in the online peer review trial. Around 7000 potential reviewers were invited to review each manuscript. In all, 44 of 110 manuscripts (40%) received 100 reviews from 59 reviewers, alongside 115 conventional reviews. The quality of the open forum reviews was lower than for conventional reviews (2.13 (± 0.75) versus 2.84 (± 0.71), P<0.001).

Conclusion
Open online peer review is feasible in this setting, but it attracts few reviews, of lower quality than conventional peer reviews.

Citation: Almquist M, von Allmen RS, Carradice D, Oosterling SJ, McFarlane K, Wijnhoven B (2017) A prospective study on an innovative online forum for peer reviewing of surgical science. PLoS ONE 12(6): e0179031. https://doi.org/10.1371/journal.pone.0179031

 View

Gender bias in scholarly peer review


Authors: Markus Helmer, Manuel Schottdorf, Andreas Neef, Demian Battaglia
Abstract: Peer review is the cornerstone of scholarly publishing and it is essential that peer reviewers are appointed on the basis of their expertise alone. However, it is difficult to check for any bias in the peer-review process because the identity of peer reviewers generally remains confidential. Here, using public information about the identities of 9000 editors and 43000 reviewers from the Frontiers series of journals, we show that women are underrepresented in the peer-review process, that editors of both genders operate with substantial same-gender preference (homophily), and that the mechanisms of this homophily are gender-dependent. We also show that homophily will persist even if numerical parity between genders is reached, highlighting the need for increased efforts to combat subtler forms of gender bias in scholarly publishing.
Citation: Helmer, M., Schottdorf, M., Neef, A., & Battaglia, D. (2017). Gender bias in scholarly peer review. eLife, 6, e21718. doi: 10.7554/eLife.21718

View

Source: Gender bias in scholarly peer review

OpenAIRE survey on open peer review: Attitudes and experience amongst editors, authors and reviewers


Authors: Tony Ross-Hellauer; Arvid Deppe; Birgit Schmidt

Abstract: Open peer review (OPR) is a cornerstone of the emergent Open Science agenda. Yet to date no large-scale survey of attitudes towards OPR amongst academic editors, authors, reviewers and publishers has been undertaken. This paper presents the findings of an online survey, conducted for the OpenAIRE2020 project during September and October 2016 that sought to bridge this information gap in order to aid the development of appropriate OPR approaches by providing evidence about attitudes towards and levels of experience with OPR. The results of this cross-disciplinary survey, which received 3,062 full responses, show the majority of respondents to be in favour of OPR becoming mainstream scholarly practice, as they also are for other areas of Open Science, like Open Access and Open Data. We also observe surprisingly high levels of experience with OPR, with three out of four (76.2%) respondents reporting having taken part in an OPR process as author, reviewer or editor. There were also high levels of support for most of the traits of OPR, particularly open interaction, open reports and final-version commenting. Respondents were against opening reviewer identities to authors, however, with more than half believing it would make peer review worse. Overall satisfaction with the peer review system used by scholarly journals seems to strongly vary across disciplines. Taken together, these findings are very encouraging for OPR’s prospects for moving mainstream but indicate that due care must be taken to avoid a “one-size fits all” solution and to tailor such systems to differing (especially disciplinary) contexts. More research is also needed. OPR is an evolving phenomenon and hence future studies are to be encouraged, especially to further explore differences between disciplines and monitor the evolution of attitudes.

Citation: Ross-Hellauer, Tony, Deppe, Arvid, & Schmidt, Birgit. (2017, May 2). OpenAIRE survey on open peer review: Attitudes and experience amongst editors, authors and reviewers. Zenodo. http://doi.org/10.5281/zenodo.570864

View

Looking into Pandora’s Box: The Content of Sci-Hub and its Usage


Author: Bastian Greshake

Abstract: Despite the growth of Open Access, potentially illegally circumventing paywalls to access scholarly publications is becoming a more mainstream phenomenon. The web service Sci-Hub is amongst the biggest facilitators of this, offering free access to around 62 million publications. So far it is not well studied how and why its users are accessing publications through Sci-Hub. By utilizing the recently released corpus of Sci-Hub and comparing it to the data of  ~28 million downloads done through the service, this study tries to address some of these questions. The comparative analysis shows that both the usage and complete corpus is largely made up of recently published articles, with users disproportionately favoring newer articles and 35% of downloaded articles being published after 2013. These results hint that embargo periods before publications become Open Access are frequently circumnavigated using Guerilla Open Access approaches like Sci-Hub. On a journal level, the downloads show a bias towards some scholarly disciplines, especially Chemistry, suggesting increased barriers to access for these. Comparing the use and corpus on a publisher level, it becomes clear that only 11% of publishers are highly requested in comparison to the baseline frequency, while 45% of all publishers are significantly less accessed than expected. Despite this, the oligopoly of publishers is even more remarkable on the level of content consumption, with 80% of all downloads being published through only 9 publishers. All of this suggests that Sci-Hub is used by different populations and for a number of different reasons, and that there is still a lack of access to the published scientific record. A further analysis of these openly available data resources will undoubtedly be valuable for the investigation of academic publishing.

Citation:  Greshake B.Looking into Pandora’s Box: The Content of Sci-Hub and its Usage.” F1000Research 2017, 6:541. (doi: 10.12688/f1000research.11366.1) .

View

OPRM: Challenges to Including Open Peer Review in Open Access Repositories


Authors: Pandelis Perakakis, Agnes Ponsati, Isabel Bernal, Carles Sierra, Nardine Osman, Concha Mosquera-de-Arancibia, and Emilio Lorenzo

Abstract: The peer review system is the norm for many publications. It involves an editor and several experts in the field providing comments for a submitted article. The reviewer remains anonymous to the author, with only the editor knowing the reviewer’s identity. This model is now being challenged and open peer review (OPR) models are viewed as the new frontier of the review process. OPR is a term that encompasses diverse variations in the traditional review process. Examples of this are modifications in the way in which authors and reviewers are aware of each other’s identity (open identities), the visibility of the reviews carried out (open reviews) or the opening up of the review to the academic community (open participation). We present the project for the implementation of an Open Peer Review Module in two major Spanish repositories, DIGITAL.CSIC and e-IEO, together with some promising initial results and challenges in the take-up process. The OPR module, designed for integration with DSpace repositories, enables any scholar to provide a qualitative and quantitative evaluation of any research object hosted in these repositories.

Citation: Perakakis, P., Ponsati, A., Bernal, I., Sierra, C., Osman, N., Mosquera-de-Arancibia, C., & Lorenzo, E. (2017). OPRM: Challenges to Including Open Peer Review in Open Access Repositories. The Code4Lib Journal, (35).

View

Source: OPRM: Challenges to Including Open Peer Review in Open Access Repositories