Data and scripts for "An Exploratory Study on Machine Learning Model Management"

Abstract

Effective model management is crucial for ensuring performance and reliability in Machine Learning (ML) systems, given the dynamic nature of data and operational environments. However, standard practices are lacking, often resulting in ad hoc approaches. To address this, our research provides a clear definition of ML model management activities, processes, and techniques. Analyzing 227 ML repositories, we propose a taxonomy of 16 model management activities and identify 12 unique challenges. We highlight documentation and bug fixing as two of the most critical model management activities. Additionally, our findings indicate a significant shift towards automation of the ML pipeline, emphasizing the adoptions of tools for data, model, and documentation versioning. To offer practical guidance, we conducted a survey with industry practitioners and academic researchers to understand how model management challenges can be addressed. Our contributions include a detailed taxonomy of model management activities, a mapping of challenges to these activities, practitioner-informed solutions for challenge mitigation, and a publicly available dataset of model management activities and challenges. This work aims to equip ML developers with knowledge and best practices essential for the robust management of ML models.