Planned intervention: On Wednesday April 3rd 05:30 UTC Zenodo will be unavailable for up to 2-10 minutes to perform a storage cluster upgrade.
Published January 31, 2019 | Version v1
Presentation Open

Collective Knowledge framework to automate, crowdsource, reproduce and share HPC experiments

  • 1. cTuning foundation, dividiti

Description

Validating experimental results from articles has finally become a norm at many HPC and systems conferences. Nowadays, more than half of accepted papers pass artifact evaluation and share related code and data. Unfortunately, lack of a common experimental framework, common research methodology and common formats places an increasing burden on evaluators to validate a growing number of ad-hoc artifacts. Furthermore, having too many ad-hoc artifacts and Docker snapshots is almost as bad as not having any (!), since they cannot be easily reused, customized and built upon.

While overviewing more than 100 papers during artifact evaluation at PPoPP, CGO, PACT and other conferences, I noticed that many of them use similar experimental setups, benchmarks, models, data sets, environments and platforms. This motivated me to develop Collective Knowledge (CK), an open workflow framework with a unified Python API to automate common researchers’ tasks such as detecting software and hardware dependencies, installing missing packages, downloading data sets and models, compiling and running programs, performing autotuning and co-design, crowdsourcing
time-consuming experiments across computing resources provided by volunteers similar to SETI@home, applying statistical analysis and machine learning, validating results and plotting them on a common scoreboard for open and fair comparison, automatically generating interactive articles, and so on: cKnowledge.org.

In this talk I will introduce CK concepts and present several real world use cases from General Motors, Amazon and Arm
on collaborative benchmarking, autotuning and co-design of efficient software/hardware stacks for deep learning. I will also present results and reusable CK components from the 1st ACM ReQuEST optimization tournament. Finally, I will introduce our latest initiative to create an open repository of reusable research components and workflows.

Notes

Presentation at the 4th EasyBuild User Meeting at Louvain-la-Neuve.

Files

fursin-talk.pdf

Files (3.9 MB)

Name Size Download all
md5:2682845768030d267c7be9ae30a8a308
3.9 MB Preview Download

Additional details

References

  • Fursin, Grigori et.al (2016). Collective Knowledge: Towards R&D sustainability.