Protocol Captures Initiative

Background

The difficulty with reproducing computational results

• Experimental studies are easy to reproduce because the methods are well-established.  Computational techniques evolve much faster: new code is written every minute, new scripts and applications automate what was previously done manually. 
• Unfortunately, this means that the complete set of code, scripts, and manual steps are too lengthy for any journal to publish. 
• Reproducing computational results for most publications requires fiddling with a source code directory and mining through home directories.

Our response: Protocol Captures

Protocol captures are the complete collection of methods employed at the time of publication to ensure that results can be reproduced with no additional coding or debugging.

Benefits

We believe that the following benefits are obtained by having all authors of articles that involve original research create a protocol capture, and encourage other computational labs to do the same:

• Sustainability: Projects can be extended with new ideas, expanded with new data, and otherwise continued with no additional coding
  • When projects are expanded or extended by the next cohort of students, they should first be able to reproduce the results of their predecessor
  • While the papers provide a broad overview of the method, reproducing the results is far from straight-forward.
  • With a protocol capture in place, new students and other people interested in the project do not need to redo all the scripting of the original authors just to be in a position to reproduce the original results
  • Encourages investigators to be keep project resources organized, simple, and efficient, as such habits will result in less work required for assembling the protocol capture
• Efficiency: Maintaining an up-to-date protocol capture while conducting computational research makes it efficient to extend the project with new data as the project evolves or takes a different direction
• Promotes scientific integrity: Protocol captures are a key component of enabling reproduction of results
  • Scientific integrity is important for the public's trust and interest in funding computational research
• Ongoing improvement
  • Many of our projects are knowledge-based and can be improved as databases and datasets grow
  • Maintaining a protocol capture makes it considerably easier to re-run these projects and update results after datasets have grown appreciably

Instructions for authors

Complete instructions on protocol capture creation