Published June 28, 2023 | Version v1
Presentation Open

Collective Mind: toward a common language to facilitate reproducible research and technology transfer

  • 1. cKnowledge
  • 2. cTuning foundation


The keynote presentation from the 1st ACM conference on reproducibility and replicability (ACM REP'23).

Video ACM YouTube channel ] [ GitHub project ] [ Related reproducibility initiatives ]

CM proof-of-concept was developed by and

During the past 10 years, we have considerably improved the reproducibility of experimental results from published papers by introducing the artifact evaluation process with a unified artifact appendix and reproducibility checklists, Jupyter notebooks, containers, and Git repositories. On the other hand, our experience reproducing more than 150 papers shows that it can take weeks and months of painful and repetitive interactions between teams to reproduce artifacts. This effort includes decrypting numerous README files, examining ad-hoc artifacts and containers, and figuring out how to reproduce computational results. Furthermore, snapshot containers pose a challenge to optimize algorithms' performance, accuracy, power consumption and operational costs across diverse and rapidly evolving software, hardware, and data used in the real world.

In this talk, I explain how our practical artifact evaluation experience and the feedback from researchers and evaluators motivated us to develop a simple, intuitive, technology agnostic, and English-like scripting language called Collective Mind (CM). It helps to automatically adapt any given experiment to any software, hardware, and data while automatically generating unified README files and synthesizing modular containers with a unified API. It is being developed by the MLCommons Task Force on Automation and Reproducibility to facilitate reproducible AI/ML Systems research and minimizing manual and repetitive benchmarking and optimization efforts, reduce time and costs for reproducible research, and simplify technology transfer to production. I also present several recent use cases of how CM helps MLCommons, the Student Cluster Competition, and artifact evaluation at ACM/IEEE conferences. I conclude with our development plans, new challenges, possible solutions, and upcoming reproducibility and optimization challenges powered by the Collective Knowledge Playground and CM:



Files (6.1 MB)

Name Size Download all
6.1 MB Preview Download

Additional details

Related works

Is new version of
Journal: 10.1098/rsta.2020.0211 (DOI)


Repository URL
Development Status