The power of open polyglot plain text tooling for reproducible AI research

Description

Abstract (En)

Methods reproducibility is the ability to record and implement all experimental and computational procedures of the experiment with the same data and tools, obtaining the same results. It is a crucial step for applied Artificial Intelligence (AI) research in an Open Science context. While AI-specific reproducibility tools exist, the typical researcher employs AI in a larger scientific context. Here, we explore synergies of existing plain-text-based, polyglot tooling to capture this whole context. First, literate programming using Org-mode allows executing, capturing and annotating all research steps. It includes research hypotheses, dataset creation, study protocols, the training and evaluation process, and the model application with analysis code and reproducible figures and papers. We detail Org-mode’s polyglot capabilities of combining arbitrary programming languages and concepts like tangling or transclusion, which integrate source and artifact files in the literate document. Second, the Nix package manager allows declaring the full polyglot environment programmatically, making it reproducible on arbitrary machines. Third, recording every change with DataLad, built on Git and git-annex, provides provenance of all results and entry points to reproduce experiments. This Open Science approach to AI experiments integrates computational reproducibility with context readability for machines and humans.