Reviewers are blinkered by bibliometrics

Authors: Paula Stephan, Reinhilde Veugelers& Jian Wang

Abstract: There is a disconnect between the research that reviewers purport to admire and the research that they actually support. As participants on multiple review panels and scientific councils, we have heard many lament researchers’ reluctance to take risks. Yet we’ve seen the same panels eschew risk and rely on bibliometric indicators for assessments, despite widespread agreement that they are imperfect measures1–6.

The review panels we observed last year were using bibliometrics in much the same way as they did before the 2015 Leiden Manifesto4, the 2012 San Francisco Declaration on Research Assessment, which Nature is signing, and similar exhortations against their use. After all, bibliometric measures offer a convenient way to help evaluate a large number of proposals and papers.

Although journal impact factors (JIFs) were developed to assess journals and say little about any individual paper, reviewers routinely justify their evaluations on the basis of where candidates have published. Panel members judge applicants by Google Scholar results and use citation counts to score proposals for new research. This practice prevails even at agencies such as the European Research Council (ERC), which instructs reviewers not to look up bibliometric measures.

As economists who study science and innovation, we see engrained processes working against cherished goals. Scientists we interview routinely say that they dare not propose bold projects for funding in part because of expectations that they will produce a steady stream of papers in journals with high impact scores. The situation may be worse than assumed. Our analysis of 15 years’ worth of citation data suggests that common bibliometric measures relying on short-term windows undervalue risky research7.

How can we move beyond declarations and wean reviewers off bibliometric indicators that bias decisions against bold work?

Citation: Paula Stephan, Reinhilde Veugelers& Jian Wang. Reviewers are blinkered by bibliometrics : Nature News & Comment. Nature 544, 411–412 (doi:10.1038/544411a


POST: Hosting the Digital Ramamala Library at Penn, or, Thinking About Open Licenses for Non-Western Digitized Manuscripts

Author: dh+lib review

Abstract: Dot Porter (University of Pennsylvania) has posted on her blog the text of a talk she gave at the Global Digital Humanities Symposium, “Hosting the Digital Rāmamālā Library at Penn, or, thinking about open licenses for non-Western digitized manuscripts.” Porter’s post traces the history of the cataloging and digitization of the Ramamala Library, “one of the oldest still-active traditional libraries in Bangladesh,” as well as the development of OPenn: Primary Digital Resources Available for Everyone. Implicit in OPenn’s design was the desire for “users of OPenn to always be certain about what they could do with the data, so we decided that anything that goes into OPenn must follow those licenses that Creative Commons has approved for Free Cultural Works… Note that licenses with a non-commercial clause are not approved for Free Cultural Works, and thus OPenn, by policy, is not able to host them.”

Citation: POST: Hosting the Digital Ramamala Library at Penn, or, Thinking About Open Licenses for Non-Western Digitized Manuscripts. dh+lib. Retrieved from


The academic, economic and societal impacts of Open Access: an evidence-based review

Authors: Jonathan P. Tennant, François Waldner, Damien C. Jacques, Paola Masuzzo, Lauren B. Collister, Chris. H. J. Hartgerink

Abstract: Ongoing debates surrounding Open Access to the scholarly literature are multifaceted and complicated by disparate and often polarised viewpoints from engaged stakeholders. At the current stage, Open Access has become such a global issue that it is critical for all involved in scholarly publishing, including policymakers, publishers, research funders, governments, learned societies, librarians, and academic communities, to be well-informed on the history, benefits, and pitfalls of Open Access. In spite of this, there is a general lack of consensus regarding the potential pros and cons of Open Access at multiple levels. This review aims to be a resource for current knowledge on the impacts of Open Access by synthesizing important research in three major areas: academic, economic and societal. While there is clearly much scope for additional research, several key trends are identified, including a broad citation advantage for researchers who publish openly, as well as additional benefits to the non-academic dissemination of their work. The economic impact of Open Access is less well-understood, although it is clear that access to the research literature is key for innovative enterprises, and a range of governmental and non-governmental services. Furthermore, Open Access has the potential to save both publishers and research funders considerable amounts of financial resources, and can provide some economic benefits to traditionally subscription-based journals. The societal impact of Open Access is strong, in particular for advancing citizen science initiatives, and leveling the playing field for researchers in developing countries. Open Access supersedes all potential alternative modes of access to the scholarly literature through enabling unrestricted re-use, and long-term stability independent of financial constraints of traditional publishers that impede knowledge sharing. However, Open Access has the potential to become unsustainable for research communities if high-cost options are allowed to continue to prevail in a widely unregulated scholarly publishing market. Open Access remains only one of the multiple challenges that the scholarly publishing system is currently facing. Yet, it provides one foundation for increasing engagement with researchers regarding ethical standards of publishing and the broader implications of ‘Open Research’.

Citation: Tennant JP, Waldner F, Jacques DC et al. The academic, economic and societal impacts of Open Access: an evidence-based review [version 3; referees: 3 approved, 2 approved with reservations]F1000Research 2016, 5:632 (doi: 10.12688/f1000research.8460.3)


Intro to HistComm Syllabus


Abstract: There is a movement happening within the discipline of history. Historians recognize that traditional scholarly communication channels (e.g., monographs, academic articles) have limited reach, appeal, and impact among non-experts.

History Communication and History Communicators (#histcomm) represent (1) the coalescing of a community of historians, journalists, media professionals and others who communicate historical scholarship to non-experts, (2) training the next generation of historians to communicate using digital and interactive media, and (3) ensuring that the ongoing work of historians contributes to public discourse by reaching students, teachers, policymakers, and audiences in innovative new ways.

Citation: Intro to HistComm Syllabus. Retrieved from


Against Capital

Author: Stuart Lawson

Abstract: The ways in which scholars exchange and share their work have evolved through pragmatic responses to the political and economic contexts in which they are embedded. So rather than being designed to fulfil their function in an optimal way, our methods of scholarly communication have been distorted by the interests of capital and by neoliberal logic. If these two interlinked political forces – that suffuse all aspects of our lives – are the reason for the mess we are currently in, then surely any alternative scholarly communication system we create should be working against them, not with them. The influence of capital in scholarly publishing, and the overwhelming force of neoliberalism in our working practices, is the problem. So when the new ‘innovative disrupters’ are fully aligned with the political forces that need to be dismantled, it is questionable that the new way of doing things is a significant improvement.

Citation: Stuart Lawson. 2017. Against Capital. ReCon: Publishing for Early Career Researchers – Immortalisation, Recognition & Metrics’. Edinburgh, Scotland. 30 June 2017


New World, Same Model: Periodicals Price Survey 2017

Authors: Stephen Bosch and Kittie Henderson

Abstract: The shifts to online and OA continue apace, but neither is causing a sea change in pricing. The shift to digital delivery of serials content has had a profound effect on the information ecosystem. Powerful discovery and social networking tools expose users to an incredibly rich world of commercially produced and open access (OA) content. Most publishers have explored new ways of pricing their content—such as population served, FTE (full-time equivalent), tiered pricing based upon Carnegie classification, or other defining criteria—or the database model, which treats all content within an e-journal package as a database, eliminating the need for title by title reconciliation. However, in the end, the pricing conversation always seems to circle back to the revenue generated by the annual subscription model.

Citation: Bosch S and Henderson K. (2017). New World, Same Model: Periodicals Price Survey 2017. Library Journal. Retrieved from


Research Access and Discovery in University News Releases: A Case Study

Author: Philip Young

Abstract: INTRODUCTION Many universities promote the peer-reviewed articles of their researchers in online news releases. However, access to the articles by the public can be limited, and information for locating articles is sometimes lacking. This exploratory study quantifies article access, the potential for immediate article archiving, and the presence of discovery aids in news releases at a large research university. METHODS A random sample of 120 news releases over an 11-year period were evaluated. RESULTS At publication, 33% of the peer-reviewed articles mentioned in news releases were open access. Immediate archiving in the institutional repository could potentially raise the access rate to 58% of the articles. Discovery aids in news releases included journal titles (96%), hyperlinks (67%), article titles (44%), and full citations (3%). No hyperlink was in the form of a referenceable digital object identifier (DOI). DISCUSSION Article availability is greater than published estimates, and could result from the university’s STEM focus or self-selection. Delayed access by journals is a significant source of availability, and provides an additional rationale for hyperlinking from news releases. CONCLUSION Most articles promoted in the university’s news releases cannot be accessed by the public. Access could be significantly increased through immediate archiving in the institutional repository. Opportunities for facilitating article discovery could increase the credibility and outreach value of news releases. Published on 2017-02-27 18:35:56

Citation: Young, P., (2017). Research Access and Discovery in University News Releases: A Case Study. Journal of Librarianship and Scholarly Communication. 5(1), p.eP2155. DOI:


The Impact Platform

Author: Jefferson Pooley

Abstract: The Conversation—”academic rigour, journalistic flair”—is the leading example of a new, web-enabled mode of academic popularization: the impact platform. The nonprofit site’s unpaid scholar-writers, together with professional staff editors, produce dozens of short, image-filled dispatches every week day. In a crucial twist, each piece is released into the web with a Creative Commons license and the hope for widescale republication. There’s no grumbling about the Huffington Post and other aggregators stealing page views: The whole point is to spread the academic news to any and all takers, as long as the author and publication are credited. The “impact” in impact platform is a nod to the motivating source for The Conversation and its imitators: the policy-driven demand for “public impact” in the Anglophone university systems. It’s no accident that The Conversation started in Australia and has its second-biggest “edition,” by far, in the UK. Both countries have adopted controversial higher-ed ranking regimes that require academics and their departments to demonstrate—and quantify—public reach. The Conversation‘s reader tallies are a convenient way to show taxpayer “return on investment.” This explains the site’s array of funders, which tend to be universities, grant-making foundations, and national research councils. The “metric tide” dynamic that underwrites the enterprise may be questionable, but the upshot is a new stage for “translated” or born-public scholarship—for all of us, not just those laboring under the Research Excellence Framework regime. Cleanly written, synoptic research capsules are ricocheting around the web and getting read. It’s spillover from the neoliberal university, and drinkable all the same.

Citation: Pooley J. 2017. The Impact Platform. Humanities Commons.


Plans and Performances: Parallels in the Production of Science and Music

Authors: David De Roure, Graham Klyne, Kevin R. Page, John Pybus, David M. Weigl, Matthew Wilcoxson, Pip Willcox

Abstract: Whether in the science lab or the music studio, we go in with a plan, we perform, and we make a record of that performance for distribution, consumption, and reuse. Both domains are increasingly data-intensive, with the adoption of new technology, and also socially intensive with democratised and growing citizen engagement. The music industry has embraced digital technology throughout the lifecycle from composition to consumption; scientific practice, and scholarly communication, are also undergoing transformation. Is the music industry more digital than science? We suggest that comparing and contrasting these two systems will provide insights of mutual benefit. Our investigation explores the notion of the Digital Music Object, analogous to the Research Object, for rich capture, sharing and reuse of both process and content.

Citation: de Roure, D, Klyne, G, Page, KR et al., (2016). Plans and performances: Parallels in the production of science and music.


ReplicationWiki: Improving Transparency in Social Sciences Research

Author: Jan H. Höffler

Abstract: In empirical social sciences research, only a small minority of study material is publicly available, therefore allowing full replication. The number of replication studies published in academic journals is even smaller. Our wiki documents the results of more than 300 replications, so far mainly in economics. It includes a database of more than 2,600 empirical studies. For each study we provide information about the availability of material for replication. This helps instructors to identify practical examples for courses focusing on empirical methods or on particular topics. Furthermore, it gives researchers better access to information on previous studies they can build on or compare their work with. We provide an overview of journals and their policies regarding data availability and publication of replications. The project has attracted interest from various fields and is open for expansion.

Citation: Jan H. Höffler. (2017). ReplicationWiki: Improving Transparency in Social Sciences Research. D-LIB Magazine, 23(3/4).