A prospective study on an innovative online forum for peer reviewing of surgical science

Authors:  Almquist M, von Allmen RS, Carradice D, Oosterling SJ, McFarlane K, Wijnhoven B

Peer review is important to the scientific process. However, the present system has been criticised and accused of bias, lack of transparency, failure to detect significant breakthrough and error. At the British Journal of Surgery (BJS), after surveying authors’ and reviewers’ opinions on peer review, we piloted an open online forum with the aim of improving the peer review process.

In December 2014, a web-based survey assessing attitudes towards open online review was sent to reviewers with a BJS account in Scholar One. From April to June 2015, authors were invited to allow their manuscripts to undergo online peer review in addition to the standard peer review process. The quality of each review was evaluated by editors and editorial assistants using a validated instrument based on a Likert scale.

The survey was sent to 6635 reviewers. In all, 1454 (21.9%) responded. Support for online peer review was strong, with only 10% stating that they would not subject their manuscripts to online peer review. The most prevalent concern was about intellectual property, being highlighted in 118 of 284 comments (41.5%). Out of 265 eligible manuscripts, 110 were included in the online peer review trial. Around 7000 potential reviewers were invited to review each manuscript. In all, 44 of 110 manuscripts (40%) received 100 reviews from 59 reviewers, alongside 115 conventional reviews. The quality of the open forum reviews was lower than for conventional reviews (2.13 (± 0.75) versus 2.84 (± 0.71), P<0.001).

Open online peer review is feasible in this setting, but it attracts few reviews, of lower quality than conventional peer reviews.

Citation: Almquist M, von Allmen RS, Carradice D, Oosterling SJ, McFarlane K, Wijnhoven B (2017) A prospective study on an innovative online forum for peer reviewing of surgical science. PLoS ONE 12(6): e0179031. https://doi.org/10.1371/journal.pone.0179031


Gender bias in scholarly peer review

Authors: Markus Helmer, Manuel Schottdorf, Andreas Neef, Demian Battaglia
Abstract: Peer review is the cornerstone of scholarly publishing and it is essential that peer reviewers are appointed on the basis of their expertise alone. However, it is difficult to check for any bias in the peer-review process because the identity of peer reviewers generally remains confidential. Here, using public information about the identities of 9000 editors and 43000 reviewers from the Frontiers series of journals, we show that women are underrepresented in the peer-review process, that editors of both genders operate with substantial same-gender preference (homophily), and that the mechanisms of this homophily are gender-dependent. We also show that homophily will persist even if numerical parity between genders is reached, highlighting the need for increased efforts to combat subtler forms of gender bias in scholarly publishing.
Citation: Helmer, M., Schottdorf, M., Neef, A., & Battaglia, D. (2017). Gender bias in scholarly peer review. eLife, 6, e21718. doi: 10.7554/eLife.21718


Source: Gender bias in scholarly peer review

OpenAIRE survey on open peer review: Attitudes and experience amongst editors, authors and reviewers

Authors: Tony Ross-Hellauer; Arvid Deppe; Birgit Schmidt

Abstract: Open peer review (OPR) is a cornerstone of the emergent Open Science agenda. Yet to date no large-scale survey of attitudes towards OPR amongst academic editors, authors, reviewers and publishers has been undertaken. This paper presents the findings of an online survey, conducted for the OpenAIRE2020 project during September and October 2016 that sought to bridge this information gap in order to aid the development of appropriate OPR approaches by providing evidence about attitudes towards and levels of experience with OPR. The results of this cross-disciplinary survey, which received 3,062 full responses, show the majority of respondents to be in favour of OPR becoming mainstream scholarly practice, as they also are for other areas of Open Science, like Open Access and Open Data. We also observe surprisingly high levels of experience with OPR, with three out of four (76.2%) respondents reporting having taken part in an OPR process as author, reviewer or editor. There were also high levels of support for most of the traits of OPR, particularly open interaction, open reports and final-version commenting. Respondents were against opening reviewer identities to authors, however, with more than half believing it would make peer review worse. Overall satisfaction with the peer review system used by scholarly journals seems to strongly vary across disciplines. Taken together, these findings are very encouraging for OPR’s prospects for moving mainstream but indicate that due care must be taken to avoid a “one-size fits all” solution and to tailor such systems to differing (especially disciplinary) contexts. More research is also needed. OPR is an evolving phenomenon and hence future studies are to be encouraged, especially to further explore differences between disciplines and monitor the evolution of attitudes.

Citation: Ross-Hellauer, Tony, Deppe, Arvid, & Schmidt, Birgit. (2017, May 2). OpenAIRE survey on open peer review: Attitudes and experience amongst editors, authors and reviewers. Zenodo. http://doi.org/10.5281/zenodo.570864


Looking into Pandora’s Box: The Content of Sci-Hub and its Usage

Author: Bastian Greshake

Abstract: Despite the growth of Open Access, potentially illegally circumventing paywalls to access scholarly publications is becoming a more mainstream phenomenon. The web service Sci-Hub is amongst the biggest facilitators of this, offering free access to around 62 million publications. So far it is not well studied how and why its users are accessing publications through Sci-Hub. By utilizing the recently released corpus of Sci-Hub and comparing it to the data of  ~28 million downloads done through the service, this study tries to address some of these questions. The comparative analysis shows that both the usage and complete corpus is largely made up of recently published articles, with users disproportionately favoring newer articles and 35% of downloaded articles being published after 2013. These results hint that embargo periods before publications become Open Access are frequently circumnavigated using Guerilla Open Access approaches like Sci-Hub. On a journal level, the downloads show a bias towards some scholarly disciplines, especially Chemistry, suggesting increased barriers to access for these. Comparing the use and corpus on a publisher level, it becomes clear that only 11% of publishers are highly requested in comparison to the baseline frequency, while 45% of all publishers are significantly less accessed than expected. Despite this, the oligopoly of publishers is even more remarkable on the level of content consumption, with 80% of all downloads being published through only 9 publishers. All of this suggests that Sci-Hub is used by different populations and for a number of different reasons, and that there is still a lack of access to the published scientific record. A further analysis of these openly available data resources will undoubtedly be valuable for the investigation of academic publishing.

Citation:  Greshake B.Looking into Pandora’s Box: The Content of Sci-Hub and its Usage.” F1000Research 2017, 6:541. (doi: 10.12688/f1000research.11366.1) .


OPRM: Challenges to Including Open Peer Review in Open Access Repositories

Authors: Pandelis Perakakis, Agnes Ponsati, Isabel Bernal, Carles Sierra, Nardine Osman, Concha Mosquera-de-Arancibia, and Emilio Lorenzo

Abstract: The peer review system is the norm for many publications. It involves an editor and several experts in the field providing comments for a submitted article. The reviewer remains anonymous to the author, with only the editor knowing the reviewer’s identity. This model is now being challenged and open peer review (OPR) models are viewed as the new frontier of the review process. OPR is a term that encompasses diverse variations in the traditional review process. Examples of this are modifications in the way in which authors and reviewers are aware of each other’s identity (open identities), the visibility of the reviews carried out (open reviews) or the opening up of the review to the academic community (open participation). We present the project for the implementation of an Open Peer Review Module in two major Spanish repositories, DIGITAL.CSIC and e-IEO, together with some promising initial results and challenges in the take-up process. The OPR module, designed for integration with DSpace repositories, enables any scholar to provide a qualitative and quantitative evaluation of any research object hosted in these repositories.

Citation: Perakakis, P., Ponsati, A., Bernal, I., Sierra, C., Osman, N., Mosquera-de-Arancibia, C., & Lorenzo, E. (2017). OPRM: Challenges to Including Open Peer Review in Open Access Repositories. The Code4Lib Journal, (35).


Source: OPRM: Challenges to Including Open Peer Review in Open Access Repositories

Peer Review and Replication Data: Best Practice from Journal of Peace Research

Authors: Nils Petter Gleditsch, Ragnhild Nordås, & Henrik Urdal

Abstract: Journal of Peace Research is an independent, interdisciplinary, and international journal devoted to the study of war and peace. It is owned by the Peace Research Institute Oslo (PRIO) and published on contract with Sage.Its articles range across all the social sciences, although a large majority of its authors now have their main training in political science. The international character of the journal is visible in the composition of the editorial committee and the authorship. The journal has long been a leader among the journals in political science and international relations in making research data publicly available, and is a pioneer in publishing dataset in the form of “special data features.”

Citation: Gleditsch, N. P., Nordås, R., & Urdal, H. (2017). Peer Review and Replication Data: Best Practice from Journal of Peace Research. College & Research Libraries, 78(3), 267–271.


Source: Peer Review and Replication Data: Best Practice from Journal of Peace Research

Open-access mega-journals: The future of scholarly communication or academic dumping ground? A review.

Authors: Valerie Spezi, Simon Wakeling, Stephen Pinfield, Claire Creaser, Jenny Fry, Peter Willett

Abstract: Open-access mega-journals (OAMJs) represent an increasingly important part of the scholarly communication landscape. OAMJs, such as PLOS ONE, are large scale, broad scope journals that operate an open access business model (normally based on article-processing charges), and which employ a novel form of peer review, focussing on scientific “soundness” and eschewing judgement of novelty or importance. The purpose of this paper is to examine the discourses relating to OAMJs, and their place within scholarly publishing, and considers attitudes towards mega-journals within the academic community.

Citation: Valerie Spezi, Simon Wakeling, Stephen Pinfield, Claire Creaser, Jenny Fry, Peter Willett. (2017) “Open-access mega-journals: The future of scholarly communication or academic dumping ground? A review”, Journal of Documentation, Vol. 73 Iss: 2, pp.263 – 283.


Source: Open-access mega-journals: The future of scholarly communication or academic dumping ground? A review: Journal of Documentation: Vol 73, No 2

Strategies and guidelines for scholarly publishing of biodiversity data

Authors: Lyubomir Penev, Daniel Mietchen, Vishwas Shravan Chavan, Gregor Hagedorn, Vincent Stuart Smith, David Shotton, Éamonn Ó Tuama, Viktor Senderov, Teodor Georgiev, Pavel Stoev, Quentin John Groom, David Remsen, Scott C. Edmunds

Abstract: The present paper describes policies and guidelines for scholarly publishing of biodiversity and biodiversity-related data, elaborated and updated during the Framework Program 7 EU BON project, on the basis of an earlier version published on Pensoft’s website in 2011. The document discusses some general concepts, including a definition of datasets, incentives to publish data and licenses for data publishing. Further, it defines and compares several routes for data publishing, namely as (1) supplementary files to research articles, which may be made available directly by the publisher, or (2) published in a specialized open data repository with a link to it from the research article, or (3) as a data paper, i.e., a specific, stand-alone publication describing a particular dataset or a collection of datasets, or (4) integrated narrative and data publishing through online import/download of data into/from manuscripts, as provided by the Biodiversity Data Journal.

The paper also contains detailed instructions on how to prepare and peer review data intended for publication, listed under the Guidelines for Authors and Reviewers, respectively. Special attention is given to existing standards, protocols and tools to facilitate data publishing, such as the Integrated Publishing Toolkit of the Global Biodiversity Information Facility (GBIF IPT) and the DarwinCore Archive (DwC-A).

A separate section describes most leading data hosting/indexing infrastructures and repositories for biodiversity and ecological data.

Citation: Penev L, Mietchen D, Chavan V, Hagedorn G, Smith V, Shotton D, Ó Tuama É, Senderov V, Georgiev T, Stoev P, Groom Q, Remsen D, Edmunds S (2017) Strategies and guidelines for scholarly publishing of biodiversity data. Research Ideas and Outcomes 3: e12431.https://doi.org/10.3897/rio.3.e12431




Single versus Double Blind Reviewing at WSDM 2017

Authors: Andrew Tomkins, Min Zhang, William D. Heavlin

Abstract: In this paper we study the implications for conference program committees of adopting single-blind reviewing, in which committee members are aware of the names and affiliations of paper authors, versus double-blind reviewing, in which this information is not visible to committee members. WSDM 2017, the 10th ACM International ACM Conference on Web Search and Data Mining, performed a controlled experiment in which each paper was reviewed by four committee members. Two of these four reviewers were chosen from a pool of committee members who had access to author information; the other two were chosen from a disjoint pool who did not have access to this information. This information asymmetry persisted through the process of bidding for papers, reviewing papers, and entering scores. Reviewers in the single-blind condition typically bid for 26% more papers, and bid preferentially for papers from top institutions. Once papers were allocated to reviewers, single-blind reviewers were significantly more likely than their double-blind counterparts to recommend for acceptance papers from famous authors and top institutions. In each case, the estimated odds multiplier is around $1.5times$, so the result is quite strong. We did not however see differences in bidding or reviewing behavior between single-blind and double-blind reviewers for papers with female authors. We describe our findings in detail and offer some recommendations.

Andrew Tomkins, Min Zhang, William D. Heavlin. (2017).  Single versus Double Blind Reviewing at WSDM 2017. arxiv