Black Box or Open Science? A study on reproducibility in AI Development Papers

The surge in Artificial Intelligence (AI) research
has spurred significant breakthroughs across various
fields. However, AI is known for its Black Box
character and reproducing AI outcomes is a
challenge. Open Science, emphasizing transparency,
reproducibility, and accessibility, is crucial in this
context, ensuring research validity and facilitating
practical AI adoption. We propose a framework to
assess the quality of AI documentation and assess 51
papers. We conclude that despite guidelines, many AI
papers fall short on reproducibility due to insufficient
documentation. It is crucial to provide comprehensive
details on training data, source code, and AI models,
and for reviewers and editors to strictly enforce
reproducibility guidelines. A dearth of detailed
methods or inaccessible source code and models can
raise questions about the authenticity of certain AI
innovations, potentially impeding their scientific
value and their adoption. Although our sample size
inhibits broad generalization, this study nonetheless
offers key insights on enhancing AI research
reproducibility.