A manifesto for reproducible science

Author: Marcus R. Munafò, Brian A. Nosek, Dorothy V. M. Bishop, Katherine S. Button, Christopher D. Chambers, Nathalie Percie du Sert, Uri Simonsohn, Eric-Jan Wagenmakers, Jennifer J. Ware & John P. A. Ioannidis

Abstract: Improving the reliability and efficiency of scientific research will increase the credibility of the published scientific literature and accelerate discovery. Here we argue for the adoption of measures to optimize key elements of the scientific process: methods, reporting and dissemination, reproducibility, evaluation and incentives. There is some evidence from both simulations and empirical studies supporting the likely effectiveness of these measures, but their broad adoption by researchers, institutions, funders and journals will require iterative evaluation and improvement. We discuss the goals of these measures, and how they can be implemented, in the hope that this will facilitate action toward improving the transparency, reproducibility and efficiency of scientific research.

Citation: Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017). A manifesto for reproducible science. Nature Human Behaviour 1. https://dx.doi.org/10.1038/s41562-016-0021


Plans and Performances: Parallels in the Production of Science and Music

Authors: David De Roure, Graham Klyne, Kevin R. Page, John Pybus, David M. Weigl, Matthew Wilcoxson, Pip Willcox

Abstract: Whether in the science lab or the music studio, we go in with a plan, we perform, and we make a record of that performance for distribution, consumption, and reuse. Both domains are increasingly data-intensive, with the adoption of new technology, and also socially intensive with democratised and growing citizen engagement. The music industry has embraced digital technology throughout the lifecycle from composition to consumption; scientific practice, and scholarly communication, are also undergoing transformation. Is the music industry more digital than science? We suggest that comparing and contrasting these two systems will provide insights of mutual benefit. Our investigation explores the notion of the Digital Music Object, analogous to the Research Object, for rich capture, sharing and reuse of both process and content.

Citation: de Roure, D, Klyne, G, Page, KR et al., (2016). Plans and performances: Parallels in the production of science and music.


Toward the Geoscience Paper of the Future: Best practices for documenting and sharing research from data to software to provenance

Authors: Gil Yolanda, Cedric H. David, Ibrahim Demir, Bakinam T. Essawy, Robinson W. Fulweiler, Jonathan L. Goodall, Leif Karlstrom, Huikyo Lee, Heath J. Mills, Ji-Hyun Oh, Suzanne A. Pierce, Allen Pope, Mimi W. Tzeng, Sandra R. Villamizar, Xuan Yu

Abstract: Geoscientists now live in a world rich with digital data and methods, and their computational research cannot be fully captured in traditional publications. The Geoscience Paper of the Future (GPF) presents an approach to fully document, share, and cite all their research products including data, software, and computational provenance. This article proposes best practices for GPF authors to make data, software, and methods openly accessible, citable, and well documented. The publication of digital objects empowers scientists to manage their research products as valuable scientific assets in an open and transparent way that enables broader access by other scientists, students, decision makers, and the public. Improving documentation and dissemination of research will accelerate the pace of scientific discovery by improving the ability of others to build upon published work.

Citation: Gil, Y., et all (2016). Toward the Geoscience Paper of the Future: Best practices for documenting and sharing research from data to software to provenance. Earth and Space Science, 3, 388-415. http://dx.doi.org/10.1002/2015EA000136 


Structuring supplemental materials in support of reproducibility

Author: Dov Greenbaum

Abstract: Supplements are increasingly important to the scientific record, particularly in genomics. However, they are often underutilized. Optimally, supplements should make results findable, accessible, interoperable, and reusable (i.e., “FAIR”). Moreover, properly off-loading to them the data and detail in a paper could make the main text more readable. We propose a hierarchical organization for supplements, with some parts paralleling and “shadowing” the main text and other elements branching off from it, and we suggest a specific formatting to make this structure explicit. Furthermore, sections of the supplement could be presented in multiple scientific “dialects”, including machine-readable and lay-friendly formats.

Citation: Greenbaum, Dov, et al., 2017.Structuring supplemental materials in support of reproducibility. Genome Biology 18:64, 10.1186/s13059-017-1205-3.


A Brief History of Archiving in Language Documentation, with an Annotated Bibliography

Authors: Ryan Henke and Andrea L. Berez-Kroeker

Abstract: We survey the history of practices, theories, and trends in archiving for the purposes of language documentation and endangered language conservation. We identify four major periods in the history of such archiving. First, a period from before the time of Boas and Sapir until the early 1990s, in which analog materials were collected and deposited into physical repositories that were not easily accessible to many researchers or speaker communities. A second period began in the 1990s, when increased attention to language endangerment and the development of modern documentary linguistics engendered a renewed and redefined focus on archiving and an embrace of digital technology. A third period took shape in the early twenty-first century, where technological advancements and efforts to develop standards of practice met with important critiques. Finally, in the current period, conversations have arisen toward participatory models for archiving, which break traditional boundaries to expand the audiences and uses for archives while involving speaker communities directly in the archival process. Following the article, we provide an annotated bibliography of 85 publications from the literature surrounding archiving in documentary linguistics. This bibliography contains cornerstone contributions to theory and practice, and it also includes pieces that embody conversations representative of particular historical periods.

Citation: Henke, Ryan and Andrea L. Berez-Kroeker. 2016. A Brief History of Archiving in Language Documentation, with an Annotated Bibliography. Language Documentation & Conservation 10. 411-457. http://hdl.handle.net/10125/24714


The Digital Archiving of Endangered Language Oral Traditions: Kaipuleohone at the University of Hawai‘i and C’ek’aedi Hwnax in Alaska

Author: Andrea L. Berez

Abstract: This essay compares and contrasts two small-scale digital endangered language archives with regard to their relevance for oral tradition research. The first is a university-based archive curated at the University of Hawai‘i, which is designed to house endangered language materials arising from the fieldwork of university researchers. The second is an indigenously-administered archive in rural Alaska that serves the language maintenance needs of the Ahtna Athabaskan Alaska Native community.

Citation: Berez, Andrea L. “The Digital Archiving of Endangered Language Oral Traditions: Kaipuleohone at the University of Hawai’i and C’ek’aedi Hwnax in Alaska.” Oral Tradition 28.2 (2013).


A survey of current reproducibility practices in linguistics publications

Authors: Gawne, Lauren; Berez-Kroeker, Andrea L.; Kelly, Barbara; Heston, Tyler

Abstract: In order to move forward toward reproducible research in linguistics, we first need to know where we are now with regard to our practices for methodological clarity and data citation in publications. In this poster we share the results of a study of over 370 journal articles, dissertations, and grammars, which is taken as a sample of current practices in the field. The publications all come from a ten-year span. The journals were selected for broad coverage. Grammars included published grammars and dissertations written as grammars, with broad geographic coverage, both in terms of subject language and publisher or university.These publications are critiqued on the basis of transparency of data source, data collection methods, analysis, and storage. While we find examples of transparent reporting, most of the surveyed research does not include key metadata, methodological information, or citations that are resolvable to the data on which the analyses are based.

Citation: Gawne, Lauren; Berez-Kroeker, Andrea L.; Kelly, Barbara; Heston, Tyler.  (2017). “A survey of current reproducibility practices in linguistics publications.” Poster presented at the Linguistic Society of America annual meeting, 5-9 January 2017, Austin TX.



Reproducible research in descriptive linguistics: integrating archiving and citation into the postgraduate curriculum at the University of Hawai’i at Mānoa

Author: Andrea L. Berez

Abstract: The notion of reproducible research has received considerable attention in recent years from physical scientists, life scientists, social and behavioural scientists, and computational scientists. Some readers will be familiar with the criterion of replicability as a tenet of good execution of the scientific method, in which sound scientific experiments or studies are those that can be recreated elsewhere leading to new data, and in which sound scientific claims are those that are confirmed by the new data in a replicated study.

Citation: Berez, A. (2015). Reproducible research in descriptive linguistics: integrating archiving and citation into the postgraduate curriculum at the University of Hawai’i at Mānoa. In A. Harris, N. Thieberger & L. Barwick (Eds.) ‘Research, records and responsibility: ten years of PARADISEC’ (pp. 39-51). Sydney: Sydney University Press.


Finding the Principles of the Commons: A Report of the Force11 Scholarly Communications Working Group

Authors: Robin Champieux, Bianca Kramer, Jeroen Bosman, Ian Bruno, Amy Buckland, Sarah Callaghan, Chris Chapman, Stephanie Hagstrom, MaryAnn E. Martone, and Daniel Paul O’Donnell


Abstract: While the creation and exchange of scholarly and research information now takes place within digital environments and increasingly on the open web, traditional print-based workflows are recapitulated across the scholarly communication life-cycle, outmoded rewards systems hold strong, and crises of access, reproducibility, and reuse continue to be raised. In some respects, scholarly and scientific communication has not changed much since the establishment of the first scientific journal 350 years ago. But, what if we could start over? What kind of system could and should we build to harnesses the resources of the digital age to maximize the communication and use of new knowledge? These questions, posed by Dr. Sarah Callaghan at the Force2015 Conference as part of the 1K Challenge, inspired the creation of the FORCE11 Scholarly Commons Working Group.


Citation: Champieux, Robin; Kramer, Bianca; Bosman, Jeroen; Bruno, Ian; Buckland, Amy; Callaghan, Sarah; Chapman, Chris; Hagstrom, Stephanie; Martone, MaryAnn E.; and O’Donnell, Daniel Paul (2016) “Finding the Principles of the Commons: A Report of the Force11 Scholarly Communications Working Group,” Collaborative Librarianship: Vol. 8 : Iss. 2 , Article 5.
Available at: http://digitalcommons.du.edu/collaborativelibrarianship/vol8/iss2/5