Preparing a new MLCommons education and reproducibility workgroup to make it easier to run, customize and reproduce MLPerf benchmarks

The 1st presentation to help prepare a new MLCommons workgroup to make it easier to run, customize and reproduce MLPerf benchmarks.

The mission:

• Develop an automated open-source workflow to make it easier to plug any real-world ML & AI tasks, models, data sets, software and hardware into the MLPerf benchmarking infrastructure.
• Use this workflow to help the newcomers learn how to customize and run MLPerf benchmarks across rapidly evolving software, hardware and data.
• Lower the barrier of entry for new MLPerf submitters and reduce their associated costs.
• Automate design space exploration of diverse ML/SW/HW stacks to trade off performance, accuracy, energy, size and costs; automate submission of Pareto-efficient configurations to MLPerf.
• Help end-users visualize all MLPerf results, reproduce them and deploy the most suitable ML/SW/HW stacks in production.
• Support reproducibility initiatives at ML and Systems conferences using rigorous MLPerf methodology and our educational toolkit.