Reproducibility and computationally intensive, data-driven research

A mini-symposium at the SIAM Conference on Computational Science & Engineering in Boston, MA on February 28, 2013.

Summary

Since Jon Claerbout adopted and started promoting reproducible research practices much has changed. While the problems for reproducibility of computational results have grown in conjunction with increases in computing power and storage densities, there has also been a steady growth in awareness of these problems and strategies to address them. In this minisymposium, we will discuss several recent attempts to come to terms with reproducibility in computational research. Topics will include education, publication, forensics and scientific integrity, as well as new technologies for provenance tracking and literate programming.

MS205 - Part I of II

Introduction (slides) and ICERM report (slides)

• K. Jarrod Millman, University of California, Berkeley, USA
• Vincent J. Carey Harvard University, USA

Reproducible Research on the Web: From Homework, Blogging to Open Journals (slides)

• Yihui Xie, Iowa State University, USA

Reproducible research used to be tied to $\LaTeX{}$ (e.g. Sweave in R) for statisticians, which has a steep learning curve and lacks many features of the web. The underlying idea of literate programming, however, is language agnostic. In this talk, we introduce the R package knitr as a general-purpose tool for reproducible research, with an emphasis on dynamic report generation on the web with Markdown, including reproducible homework, blog posts and online journals in statistics.

Rethinking How we Work with Documents

• Duncan W. Temple Lang, University of California, Davis, USA

Reproducible research tools require capturing all of the different avenues and lines of exploration in research. The document should be a database that can be rendered in different ways for different audiences, allowing dynamic results replacing code, enabling reader interactivity to explore different approaches and what-ifs. We also want to be able to programmatically query, update and verify this document database. All of this leads us to a different structure and approach for authoring documents.

Reproducible Research in Graduate Education in the Computational Sciences

• Sorin Mitran, University of North Carolina at Chapel Hill, USA

Instilling good habits of reproducible computational research can be woven throughout graduate student education. Current software tools allow drafting of live documents that contain theoretical derivations, generation of computational code, verifiable execution of code, and preparation of reports. The talk presents experience in this approach in graduate courses at UNC.

MS224 - Part II of II

Publishing Reproducible Research: Thoughts on Journal Policy

• Roger Peng, Johns Hopkins Bloomberg School of Public Health

I will also discuss the role that journals can play in encouraging reproducible research and will review the recent reproducibility policy at the journal Biostatistics.

Disseminating Reproducible Computational Research: Tools, Innovations, and Best Practices (slides)

• Victoria Stodden, Columbia University, USA

Computation is now widely recognized as central to the scientific enterprise, and numerous efforts are emerging to incorporate code and data sharing into standards of research dissemination. This goal is challenging from a number of perspectives, including effective research practices. In this talk I discuss novel innovations and best practices for facilitating code and data sharing, both at the time of publication and during the research itself, that support the underlying rational of reproducible research.

A Portrait of One Scientist as a Graduate Student

• Paul Ivanov, Redwood Center for Theoretical Neuroscience, University of California, Berkeley (slides)

In this talk, I will focus on the how of reproducible research. I will focus on specific tools and techniques I have found invaluable in doing research in a reproducible manner. In particular, I will cover the following general topics (with specific examples in parentheses): version controland code provenance (git), code verification (test driven development, nosetests), data integrity (sha1, md5, git-annex), seed saving ( random seed retention ) distribution of datasets (mirroring, git-annex, metalinks), light-weight analysis capture ( ttyrec, ipython notebook)

Reproducible Research and Omics: Thoughts from the IOM Review

• Keith A. Baggerly, Department of Bioinformatics and Computational Biology, UT MD Anderson Cancer Center (slides)

Between 2007 and 2010, several genomic “signatures” were used to guide patient therapy in clinical trials in cancer. Unfortunately, the signatures were wrong, and trials proceeded despite warnings to this effect. The Institute of Medicine (IOM) subsequently reviewed the level of evidence that should be required in such situations. Many recommendations focus on reproducibility and data integrity, including directives to funders, journals, and regulatory agencies. We briefly review the report and implications for reproducible research.