[Limor Peer]

The TRACE project is an NSF-funded project working on development mechanism and tools to support transparency and trust in computational research. It is a joint project between University of Illinois, the Odum Institute, and Cornell University.

The TRACE project has provided additional information on their vision for reproducibility and its challenges in the thread below. Please feel free to continue the conversation within the thread!

Briefly, tell us about your work/organization and how it’s related to computational reproducibility; What are you trying to address and how?

Our motivating question is the following:

How can we trust the integrity of results from research that relies on computations without repeating them?

Research communities across the sciences face a conundrum: to ensure the transparency and reproducibility of computational results, they require that authors share the data, code, and methods used to obtain them. However, without verification by repeating the computations, there is no guarantee that the author-provided artifacts are complete or can actually be used to reproduced results.

Particularly problematic are studies that employ sensitive or proprietary data for which access and reuse are restricted; streaming, transient, or ephemeral data that cannot be used to verify reproducibility due to their dynamic nature; or very large-scale or specialized computational resources available only to authorized users. In these cases, verification by repeating computations may not be possible.

TRACE presents a solution to this problem: certify the successful original execution of the computational workflow that produced the reported findings in situ. We call this certified transparency—a trustworthy record of computations signed by the systems within which they were performed.

What is your/your organization’s vision when it comes to computational reproducibility (e.g., all scholarship is computationally reproducible by default)?

Our vision is to increase trust in complex computational science through transparency, and by leveraging trust and credibility mechanisms where those exist. Even if we might not trust individual researchers, we may place higher (if not absolute) trust in government research centers or entire universities that enable those computations and data access. Developing a mechanism so that the trust chain behind computational science – and the limitations thereof – is clear, and expressive, is our goal.

What are some of the challenges you see to achieving this vision?

Challenges arise in the particular vast diversity of computational methods, and computational infrastructure. We hope that our proposed mechanism – metadata structure and helper tools – is sufficiently flexible to be deployable in a large variety of environments and institutions. Institutions, not only individual researchers, are key to making this work, but are also slower to adapt.

What would you like to ask the members of our Interest Group?

If you are curious and interested in collaborating, please reach out to anybody on our team, see https://transparency-certified.github.io/#contact for contact methods.

Discussion