2024-03-29T14:57:10Z
https://zenodo.org/oai2d
oai:zenodo.org:4013289
2020-09-04T00:59:24Z
user-ck
Grigori Fursin
2020-09-03
<p>This is software documentation for the <a href="https://github.com/ctuning/ck">Collective Knowledge framework</a> v1.15.0.</p>
<p>Related resources:</p>
<ul>
<li><a href="https://doi.org/10.5281/zenodo.4005773">"Enabling reproducible ML and systems research: the good, the bad, and the ugly"</a> (invited talk at FastPath'20 at ISPASS'20)</li>
<li><a href="https://doi.org/10.5281/zenodo.4005600">"MILEPOST Project Experience: building ML-based self-optimizing compiler"</a> (invited talk at Google compile+ML seminar)</li>
<li><a href="https://arxiv.org/abs/2006.07161">"The Collective Knowledge project: making ML models more portable and reproducible with open APIs, reusable best practices and MLOps"</a> (arXiv:2006.07161)</li>
</ul>
https://doi.org/10.5281/zenodo.4013289
oai:zenodo.org:4013289
eng
Zenodo
https://zenodo.org/communities/ck
https://doi.org/10.5281/zenodo.4013288
info:eu-repo/semantics/openAccess
Creative Commons Attribution 4.0 International
https://creativecommons.org/licenses/by/4.0/legalcode
collective knowledge
fair principles
automation
reproducibility
reusability
database
workflow
elasticsearch
common api
best practices
devops
mlops
knowledge management
reusable components
portable workflows
json
api
mlperf
open science
Collective Knowledge
info:eu-repo/semantics/technicalDocumentation
oai:zenodo.org:2554011
2020-01-20T17:10:33Z
openaire
user-ck
Grigori Fursin
2019-01-31
<p>Validating experimental results from articles has finally become a norm at many HPC and systems conferences. Nowadays, more than half of accepted papers pass artifact evaluation and share related code and data. Unfortunately, lack of a common experimental framework, common research methodology and common formats places an increasing burden on evaluators to validate a growing number of ad-hoc artifacts. Furthermore, having too many ad-hoc artifacts and Docker snapshots is almost as bad as not having any (!), since they cannot be easily reused, customized and built upon.</p>
<p>While overviewing more than 100 papers during artifact evaluation at PPoPP, CGO, PACT and other conferences, I noticed that many of them use similar experimental setups, benchmarks, models, data sets, environments and platforms. This motivated me to develop <a href="https://github.com/ctuning/ck">Collective Knowledge (CK)</a>, an open workflow framework with a unified Python API to automate common researchers’ tasks such as detecting software and hardware dependencies, installing missing packages, downloading data sets and models, compiling and running programs, performing autotuning and co-design, crowdsourcing<br>
time-consuming experiments across computing resources provided by volunteers similar to SETI@home, applying statistical analysis and machine learning, validating results and plotting them on a common scoreboard for open and fair comparison, automatically generating interactive articles, and so on: <a href="http://cKnowledge.org">cKnowledge.org</a>.</p>
<p>In this talk I will introduce CK concepts and present several real world use cases from General Motors, Amazon and Arm<br>
on collaborative benchmarking, autotuning and co-design of efficient software/hardware stacks for deep learning. I will also present results and reusable CK components from the <a href="http://cKnowledge.org/request">1st ACM ReQuEST optimization tournament</a>. Finally, I will introduce our latest initiative to create an open repository of reusable research components and workflows.</p>
Presentation at the 4th EasyBuild User Meeting at Louvain-la-Neuve.
https://doi.org/10.5281/zenodo.2554011
oai:zenodo.org:2554011
eng
Zenodo
https://zenodo.org/communities/ck
https://doi.org/10.5281/zenodo.2554010
info:eu-repo/semantics/openAccess
Creative Commons Attribution 4.0 International
https://creativecommons.org/licenses/by/4.0/legalcode
collective knowledge
open science
collaborative research
reproducible research
reusable components
common APIs
abstraction
portable workflows
native environment
Collective Knowledge framework to automate, crowdsource, reproduce and share HPC experiments
info:eu-repo/semantics/lecture
oai:zenodo.org:2422877
2018-12-19T14:30:32Z
user-ck
Todd Gamblin
Anton Lokhmotov
Kenneth Hoste
Damian Alvarez
Grigori Fursin
Flavio Vella
Stephen Herbein
Carsten Uphoff
Michael Bader
2018-12-19
<p>This repository contains a beta <a href="https://github.com/ctuning/ck">Collective Knowledge</a> workflow to automate installation, execution, customization and validation of the SeisSol application from the <a href="https://sc18.supercomputing.org/sc18-announces-selected-paper-for-next-student-cluster-competition-reproducibility-challenge">Student Cluster Competition Reproducibility Challenge at Supercomputing'18</a><br>
across different platforms, environments and datasets. It supports our <a href="http://cTuning.org/ae">long-term community initiative</a> to improve reproducibility of experimental results from published papers, automate artifact evaluation, and help the community share code, data and workflows as reusable components!</p>
<p>Documentation: <a href="https://github.com/ctuning/ck-scc18/wiki">github.com/ctuning/ck-scc18/wiki</a></p>
<p> </p>
https://doi.org/10.5281/zenodo.2422877
oai:zenodo.org:2422877
Zenodo
https://doi.org/10.1145/3126908.3126948
https://zenodo.org/communities/ck
https://doi.org/10.5281/zenodo.2422876
info:eu-repo/semantics/openAccess
BSD 3-Clause "New" or "Revised" License
https://opensource.org/licenses/BSD-3-Clause
Collective Knowledge Workflow
Student Cluster Competition
Reproducibility Challenge
Supercomputing'18
Artifact Evaluation
Research Automation
Artifact Reusability
CK workflow: SeisSol application from the SCC Reproducibility Challenge at Supercomputing'18
info:eu-repo/semantics/other
oai:zenodo.org:2455637
2020-01-25T07:21:15Z
software
user-ck
Grigori Fursin
Anton Lokhmotov
Dmitry Savenko
Eben Upton
2018-12-20
<p>This repository contains a snapshot of a <a href="https://github.com/ctuning/ck">Collective Knowledge</a> workflow with a reproducible article for collaborative research into multi-objective autotuning and machine learning techniques. It supports our <a href="http://ctuning.org/ae">long-term community initiative</a> to improve reproducibility of experimental results from published papers, automate artifact evaluation, and help the community share code, data and workflows as reusable components! </p>
<p>You can find more details about our approach in this reproducible and interactive article automatically generated by this CK workflow: <a href="http://cKnowledge.org/rpi-crowd-tuning">http://cKnowledge.org/rpi-crowd-tuning</a> . Shared results from crowd-tuning across diverse devices provided by volunteers are available in a reproducible form in this repository: <a href="http://cknowledge.org/repo-beta">http://cknowledge.org/repo-beta</a> .</p>
<p> </p>
https://doi.org/10.5281/zenodo.2455637
oai:zenodo.org:2455637
Zenodo
https://arxiv.org/abs/1801.08024
https://zenodo.org/communities/ck
https://doi.org/10.5281/zenodo.2455636
info:eu-repo/semantics/openAccess
Creative Commons Attribution 4.0 International
https://creativecommons.org/licenses/by/4.0/legalcode
Collective Knowledge workflow
Reproducible article
Interactive article
machine learning
autotuning
crowd-tuning
llvm
gcc
knowledge repository
experiment crowdsourcing
artifact evaluation
artifact reusability
portable workflows
reproducible research
open science
CK workflow and reproducible article: "A Collective Knowledge workflow for collaborative research into multi-objective autotuning and machine learning techniques"
info:eu-repo/semantics/other
oai:zenodo.org:3865242
2020-05-30T22:18:21Z
openaire
user-ck
Grigori Fursin
2020-05-29
<p>Developing novel applications based on deep tech (ML, AI, HPC, quantum, IoT) and deploying them in production is a very painful, ad-hoc, time consuming and expensive process due to continuously evolving software, hardware, models, data sets and research techniques.</p>
<p>After struggling with these problems for many years, I started the <strong>Collective Knowledge project (CK)</strong> to decompose complex systems and research projects into <a href="https://cKnowledge.io/browse">reusable, portable, customizable and non-virtualized CK components</a> with unified <a href="https://cKnowledge.io/actions">automation actions, Python APIs, CLI and JSON meta descriptions.</a></p>
<p>My idea is to gradually abstract all existing artifacts (software, hardware, models, data sets, results) and use the DevOps methodology to connect such components together into <a href="https://cKnowledge.io/demo">functional CK solutions</a>. Such solutions can automatically adapt to evolving models, data sets and bare-metal platforms with the help of <a href="https://cKnowledge.io/programs">customizable program workflows</a>, a list of <a href="https://cknowledge.io/solution/demo-obj-detection-coco-tf-cpu-benchmark-linux-portable-workflows/#dependencies">all dependencies</a> (models, data sets, frameworks), and a portable <a href="https://cKnowledge.io/packages">meta package manager</a>.</p>
<p>CK is basically our intermediate language to connect researchers and practitioners to collaboratively design, benchmark, optimize and validate innovative computational systems. It then makes it possible to find the most efficient system configutations on a <a href="https://cKnowledge.org/request.html">Pareto frontier</a> (trading off speed, accuracy, energy, size and different costs) using an <a href="https://cKnowledge.io">open repository of knowledge</a> with <a href="https://cKnowledge.io/results">live SOTA scoreboards</a> and <a href="https://cKnowledge.io/reproduced-papers">reproducible papers</a>.</p>
<p><em>Even though the <a href="https://github.com/ctuning/ck">CK technology</a> is used <a href="https://cKnowledge.org/partners.html">in production</a> for more than 5 years, it is still a proof-of-concept prototype requiring further improvements and standardization. We plan to develop a user-friendly web front-end to make it easier for researchers and practitioners to create and share CK workflows, artifacts, SOTA scoreboards, live papers, and participate in collaborative, trustable and reproducible R&D.</em></p>
https://doi.org/10.5281/zenodo.3865242
oai:zenodo.org:3865242
eng
Zenodo
https://zenodo.org/communities/ck
https://doi.org/10.5281/zenodo.3865241
info:eu-repo/semantics/openAccess
Creative Commons Attribution 4.0 International
https://creativecommons.org/licenses/by/4.0/legalcode
workflow
portability
reproducibility
adaptability
live paper
crowd-benchmarking
devops
mlops
reproducible-research
sota
sota scoreboard
automation
automation actions
reusability
crowd-tuning
machine learning
The Collective Knowledge project: closing the gap between ML&systems research and practice with portable workflows, reusable automation actions, and reproducible crowd-benchmarking
info:eu-repo/semantics/lecture
oai:zenodo.org:2555622
2020-01-25T07:27:27Z
user-artifact-evaluation
software
user-ck
Anton Lokhmotov
Grigori Fursin
2019-02-02
<p>This is a windows installer for the open-source <a href="http://cKnowledge.org">Collective Knowledge framework</a> (BSD license). It also includes all required components to use CK: Git 2.20.1 (GPL license) and Python 3.7.2 (Python Software Foundation License v2).</p>
<p>Just unzip this archive to any dedicated directory and run one of the following two scripts:</p>
<p>1) install-pip.bat to install CK via PIP</p>
<p>2) install-github.bat to install CK from <a href="https://github.com/ctuning/ck">GitHub</a></p>
<p>These scripts will install Python in your dedicated directory and will ask you to add several environment variables to your system (just copy/paste them) - that's all!</p>
<p> </p>
https://doi.org/10.5281/zenodo.2555622
oai:zenodo.org:2555622
Zenodo
https://zenodo.org/communities/artifact-evaluation
https://zenodo.org/communities/ck
https://doi.org/10.5281/zenodo.2555621
info:eu-repo/semantics/openAccess
BSD 3-Clause "New" or "Revised" License
https://opensource.org/licenses/BSD-3-Clause
API
Python API
JSON API
system abstraction
portable workflows
adaptive workflows
reproducible research
open science
collaborative experiments
Windows installer for the Collective Knowledge framework with Git 2.20.1 and Python 3.7.2
info:eu-repo/semantics/other
oai:zenodo.org:2271498
2019-06-22T09:55:27Z
user-ck
The TensorFlow Authors
2018-12-14
<p>MobileNet models for <a href="https://www.tensorflow.org">TensorFlow </a>archived from <a href="https://github.com/tensorflow/models/blob/master/research/slim/nets/mobilenet_v1.md">github.com/tensorflow/models/blob/master/research/slim/nets/mobilenet_v1.md</a> to ensure the stability of <a href="https://github.com/ctuning/ck">Collective Knowledge</a> <a href="https://ReproIndex.com/components/&c=package">packages</a> and <a href="https://github.com/ctuning/ck-mlperf">workflows</a> used to automate the <a href="https://mlperf.org">MLPerf benchmark</a>.</p>
<p><a href="https://github.com/tensorflow/models/blob/master/LICENSE">Copyright The TensorFlow Authors. Apache License Version 2.0</a><br>
</p>
https://doi.org/10.5281/zenodo.2271498
oai:zenodo.org:2271498
Zenodo
https://arxiv.org/abs/1704.04861
https://zenodo.org/communities/ck
https://doi.org/10.5281/zenodo.2271496
info:eu-repo/semantics/openAccess
Apache License 2.0
http://www.apache.org/licenses/LICENSE-2.0
MLPerf
Model
inference
MobileNets
TensorFlow
v1
mobilenet-v1-.tensorflow model
info:eu-repo/semantics/other
oai:zenodo.org:2544262
2020-01-20T17:44:50Z
openaire
user-ck
Grigori Fursin
2018-11-01
<p>The original presentation was shared via <a href="https://www.slideshare.net/GrigoriFursin/accelerating-open-science-and-ai-with-automated-portable-customizable-and-reusable-research-components-and-workflows">SlideShare</a>.</p>
<p>Validating experimental results from articles has finally become a norm at many systems and ML conferences. Nowadays, more than half of accepted papers pass artifact evaluation and share related code and data. Unfortunately, lack of a common experimental framework, common research methodology, and common formats places an increasing burden on evaluators to validate a growing number of ad-hoc artifacts. Furthermore, having too many ad-hoc artifacts and Docker snapshots is almost as bad as not having any (!), since they cannot be easily reused, customized and built upon.<br>
<br>
While overviewing more than 100 papers during artifact evaluation at PPoPP, CGO, PACT, Supercomputing, and other conferences, we noticed that many of them use similar experimental setups, benchmarks, models, data sets, environments and platforms. This motivated us to develop Collective Knowledge (CK), an open workflow framework with a unified Python API to automate common researchers’ tasks such as detecting software and hardware dependencies, installing missing packages, downloading data sets and models, compiling and running programs, performing autotuning and co-design, crowdsourcing time-consuming experiments across computing resources provided by volunteers similar to SETI@home, applying statistical analysis and machine learning, validating results and plotting them on a common scoreboard for open and fair comparison, automatically generating interactive articles, and so on: <a href="http://cKnowledge.org">cKnowledge.org</a>.<br>
<br>
In this presentation, we will introduce CK concepts and present several real-world use cases from General Motors and Arm<br>
on collaborative benchmarking, autotuning and co-design of efficient software/hardware stacks for deep learning. We also present results and reusable CK components from the 1st ACM ReQuEST optimization tournament: http://cKnowledge.org/request. Finally, we introduce our latest initiative to create an open repository of reusable research components and workflows to reboot and accelerate open science, quantum computing, and AI!</p>
https://doi.org/10.5281/zenodo.2544262
oai:zenodo.org:2544262
eng
Zenodo
https://doi.org/10.3850/9783981537079_1018
https://arxiv.org/abs/arXiv:1801.06378
https://arxiv.org/abs/arXiv:1407.3487
https://zenodo.org/communities/ck
https://doi.org/10.5281/zenodo.2544261
info:eu-repo/semantics/openAccess
Creative Commons Attribution 4.0 International
https://creativecommons.org/licenses/by/4.0/legalcode
open science
artificial intelligence
automated workflows
portable workflows
reusable research components
reusable research
collaborative research
collaborative learning
Accelerating open science and AI with automated, portable, customizable and reusable research components and workflows
info:eu-repo/semantics/lecture
oai:zenodo.org:4005588
2020-11-23T22:43:50Z
openaire
user-ck
Grigori Fursin
2020-08-28
<p><strong>Invited talk at FastPath 2020 (International Workshop on Performance Analysis of Machine Learning Systems) co-located with ISPASS 2020.</strong></p>
<p><strong>Program: </strong><a href="https://fastpath2020.github.io/Program">fastpath2020.github.io/Program</a></p>
<p><strong>Abstract:</strong></p>
<p>10 years ago we released our novel ML-based MILEPOST compiler with all the related experimental data at <a href="http://ctuning.org">cTuning.org</a>. Unfortunately, this research quickly stalled after we struggled to reproduce performance results and predictive models shared by volunteers across rapidly changing systems.</p>
<p>In this talk, I will describe my 10-year effort to solve numerous reproducibility issues in ML&systems research. I will share my experience reproducing 150+ systems and ML papers during artifact evaluation at ASPLOS, MLSys, CGO, PPoPP and Supercomputing. This tedious experience motivated me to develop the <a href="http://cknowledge.org">cKnowledge.org</a> framework and the open <a href="https://cKnowledge.io">cKnowledge.io</a> portal to bring DevOps principles to our research. I will also present cKnowledge solutions - a new way to package and share research artifacts and results with common Python APIs, CLI actions, portable workflows and JSON meta descriptions. Such solutions can be used to automatically build, benchmark and validate ML&system experiments across continuously evolving platforms.</p>
<p>I will conclude with <a href="https://cKnowledge.org/partners">several practical use-cases</a> of our technology in collaboration with Arm, IBM, General Motors, the Raspberry Pi foundation and MLPerf. Our long-term goal is to help researchers share their new ML techniques as production-ready packages along with published papers and participate in collaborative and reproducible benchmarking, co-design and comparison of efficient ML/software/hardware stacks.</p>
<p> </p>
https://doi.org/10.5281/zenodo.4005588
oai:zenodo.org:4005588
eng
Zenodo
https://zenodo.org/communities/ck
https://doi.org/10.5281/zenodo.4005587
info:eu-repo/semantics/openAccess
Creative Commons Attribution 4.0 International
https://creativecommons.org/licenses/by/4.0/legalcode
reproducibility
reusability
portability
workflow
automation
machine learning
systems
benchmarking
artifact evaluation
fair principles
open science
Enabling reproducible ML&systems research: the good, the bad, and the ugly
info:eu-repo/semantics/lecture
oai:zenodo.org:2266646
2019-04-10T02:10:14Z
user-ck
The TensorFlow Authors
2018-12-14
<p>MobileNet models for <a href="https://www.tensorflow.org">TensorFlow </a>archived from <a href="https://github.com/tensorflow/models/blob/master/research/slim/nets/mobilenet">github.com/tensorflow/models/blob/master/research/slim/nets/mobilenet</a> (<a href="https://storage.googleapis.com/mobilenet_v2/checkpoints">storage.googleapis.com/mobilenet_v2/checkpoints</a>) to ensure the stability of Collective Knowledge <a href="http://cKnowledge.org/shared-packages.html">packages</a> and <a href="https://github.com/ctuning/ck-mlperf">workflows</a> used to automate the <a href="https://mlperf.org">MLPerf benchmark</a>.</p>
<p><a href="https://github.com/tensorflow/models/blob/master/LICENSE">Copyright The TensorFlow Authors. Apache License Version 2.0</a><br>
</p>
https://doi.org/10.5281/zenodo.2266646
oai:zenodo.org:2266646
Zenodo
https://arxiv.org/abs/1801.04381
https://zenodo.org/communities/ck
https://doi.org/10.5281/zenodo.2266645
info:eu-repo/semantics/openAccess
Apache License 2.0
http://www.apache.org/licenses/LICENSE-2.0
MLPerf
Model
inference
MobileNets
TensorFlow
v2
MobileNets-v2-20181214 for MLPerf Inference (Edge)
info:eu-repo/semantics/other
oai:zenodo.org:2544230
2020-01-20T17:40:18Z
user-artifact-evaluation
openaire
user-ck
Grigori Fursin
2013-10-11
<p>Continuing innovation in science and technology is vital for our society and requires ever-increasing computational resources. However, delivering such resources has become intolerably complex, ad-hoc, costly and error-prone due to an enormous number of available design and optimization choices combined with the complex interactions between all software and hardware components. Auto-tuning, run-time adaptation, and machine learning based approaches have been demonstrating good promise to address above challenges for more than a decade but are still far from the widespread production use due to unbearably long exploration and training times, lack of a common experimental methodology, and lack of public repositories for unified data collection, analysis and mining.<br>
<br>
In this talk, I presented a long-term holistic and cooperative methodology and infrastructure for systematic characterization and optimization of computer systems through unified and scalable repositories of knowledge and crowdsourcing. In this approach, multi-objective program and architecture tuning to balance performance, power consumption, compilation time, code size and any other important metric are transparently distributed among multiple users while utilizing any available mobile, cluster or cloud computing services. Collected information about program and architecture properties and behavior is continuously processed using statistical and predictive modeling techniques to build, keep and share only useful knowledge at multiple levels of granularity. Gradually increasing and systematized knowledge can be used to predict most profitable program optimizations, run-time adaptation scenarios and architecture configurations depending on user requirements. I also presented a new version of the public, open-source infrastructure and repository (cTuning3 aka Collective Mind) for crowdsourcing auto-tuning using thousands of shared kernels, benchmarks and datasets combined with online learning plugins. Finally, I also discussed encountered challenges and some future collaborative research directions on the way towards Exascale computing.</p>
https://doi.org/10.5281/zenodo.2544230
oai:zenodo.org:2544230
eng
Zenodo
https://arxiv.org/abs/arXiv:1407.3487
https://zenodo.org/communities/artifact-evaluation
https://zenodo.org/communities/ck
https://doi.org/10.5281/zenodo.2544229
info:eu-repo/semantics/openAccess
Creative Commons Attribution 4.0 International
https://creativecommons.org/licenses/by/4.0/legalcode
open science
collaborative research
efficient systems
crowdsourcing autotuning
self-optimizing systems
brain-inspired computing
cTuning.org: systematizing tuning of computer systems using crowdsourcing and statistics
info:eu-repo/semantics/lecture
oai:zenodo.org:2555725
2019-02-02T13:18:03Z
user-ck
user-eu
Grigori Fursin
2019-02-02
<p>Experimental results in the open <a href="http://cKnowledge.org">Collective Knowledge format</a> for GCC and LLVM compiler flags autotuning crowdsourced across diverse Android devices provided by volunteers:</p>
<ul>
<li><a href="http://cKnowledge.org/rpi-crowd-tuning">Interactive paper automatically generated via CK</a></li>
<li><a href="https://arxiv.org/abs/1801.08024">ArXiv paper</a></li>
<li><a href="http://cknowledge.org/android-apps.html">Android app</a></li>
<li><a href="http://cknowledge.org/gcc-crowd-benchmarking-results">Public dashboard for GCC</a></li>
<li><a href="http://cknowledge.org/llvm-crowd-benchmarking-results">Public dashboard for LLVM</a></li>
</ul>
<p>You can install those results in your local CK repository as follows:</p>
<pre><code>ck pull repo:ck-crowdtuning
ck unzip repo --zip=ckr-upload-gcc-llvm-flags.20190201.zip
</code></pre>
<p> </p>
https://doi.org/10.5281/zenodo.2555725
oai:zenodo.org:2555725
Zenodo
https://zenodo.org/communities/ck
https://zenodo.org/communities/eu
https://doi.org/10.5281/zenodo.2555724
info:eu-repo/semantics/openAccess
Creative Commons Attribution 4.0 International
https://creativecommons.org/licenses/by/4.0/legalcode
collective knowledge
compiler crowd-tuning
compiler autotuning
crowdsourcing experiments
open science
collaborative research
Experimental results in the CK format (GCC and LLVM compiler flag crowd-tuning)
info:eu-repo/semantics/other
oai:zenodo.org:3908799
2020-06-26T10:18:23Z
user-artifact-evaluation
openaire
user-ck
Grigori Fursin
2017-03-14
<p><em>14 March 2017, CNRS webinar, Grenoble, France</em></p>
<p>A decade ago <a href="http://fursin.net/research.html">my research</a> nearly stalled. I was investigating how to crowdsource performance analysis and optimization of realistic workloads across diverse hardware provided by volunteers and combine it with machine learning [1]. Often, it was simply impossible to reproduce crowdsourced empirical results and build predictive models due to continuously changing software and hardware stacks. Worse still, lack of realistic workloads and representative data sets in our community severely limited the usefulness of such models.<br>
<br>
All these problems motivated me to create a public portal (<a href="http://cTuning.org">cTuning.org</a>) to share, validate and reuse workloads, data sets, tools, experimental results, and predictive models while involving the community in this effort [2]. This experience, in turn, helped us to initiate the so-called <a href="http://ctuning.org/ae">Artifact Evaluation</a> (AE) at ACM conferences on parallel programming, architecture and code generation (ASPLOS, CGO, PPoPP, PACT, SC and MLSys). AE aims to independently validate experimental results reported in the publications and to encourage code and data sharing.</p>
<p>These slides are from my webinar <a href="https://github.com/alegrand/RR_webinars/blob/master/8_artifact_evaluation/index.org"><em>“Enabling open and reproducible research at computer systems conferences: the good, the bad and the ugly”</em></a> at CNRS Grenoble (14 March 2017). I shared my practical experience organizing Artifact Evaluation over the past years, along with encountered problems and possible solutions.<br>
<br>
On the one hand, we have received incredible support from the research community, <a href="http://acm.org/">ACM</a>, universities, and companies. We have even received a record number of artifact submissions at the <a href="http://ctuning.org/ae">CGO/PPoPP'17</a> AE (27 vs 17 two years ago) sponsored by <a href="http://research.nvidia.com/">NVIDIA</a> and the <a href="http://ctuning.org/">cTuning foundation</a>. We have also introduced <a href="https://github.com/ctuning/ck-artifact-evaluation/blob/master/wfe/artifact-evaluation/templates/ae-20190108.tex">Artifact Appendices</a> and co-authored the new <a href="http://www.acm.org/publications/policies/artifact-review-badging">ACM Result and Artifact Review and Badging</a> policy now used at <a href="http://sc17.supercomputing.org/submitters/technical-papers/reproducibility-initiatives-for-technical-papers">Supercomputing</a>. <br>
<br>
On the other hand, the use of proprietary benchmarks, rare hardware platforms, and totally ad-hoc scripts to set up, run and process experiments all place a huge burden on evaluators. It is simply too difficult and time-consuming to customize and rebuild experimental setups, reuse artifacts and eventually build upon others’ efforts - the main pillars of open science!<br>
<br>
I then present <a href="http://cknowledge.org/">Collective Knowledge (CK)</a>, my attempt to introduce a customizable workflow framework with a unified JSON API and a <a href="https://github.com/ctuning/ck/wiki/Portable-workflows">cross-platform package manager</a>, that can automate ML&systems R&D and enable live papers while automatically adapting to continuously evolving software and hardware [3]. I also demonstrate a practical CK workflow to <a href="http://cknowledge.org/ai">collaboratively optimize deep learning</a> across different compilers, libraries, data sets and diverse platforms from resource-constrained mobile devices to data centers (see our <a href="http://cknowledge.org/android-apps.html">Android app to crowdsource DNN optimization</a> across diverse mobile devices provided by volunteers, and the <a href="http://cknowledge.org/repo-beta">public repository with results</a>) [4].<br>
<br>
Finally, I describe our novel publication model to reproduce results from published papers with the help of the community [5]. <br>
<br>
<em>Please feel free to contact me at <a href="mailto:Grigori.Fursin@cTuning.org">Grigori.Fursin@cTuning.org</a> if you have any questions or comments! I am looking forward to your feedback! </em></p>
<p><strong>References</strong></p>
<ol>
<li><a href="https://scholar.google.com/citations?view_op=view_citation&hl=en&user=IwcnpkwAAAAJ&citation_for_view=IwcnpkwAAAAJ:LkGwnXOMwfcC">“Milepost GCC: Machine learning enabled self-tuning compiler”, International journal of parallel programming</a>, Volume 39, Issue 3, pp.296-327, 2009</li>
<li><a href="https://hal.inria.fr/inria-00436029v2">“Collective Tuning Initiative: automating and accelerating development and optimization of computing systems”</a>, GCC Developers' Summit, Montreal, Canada. 2009</li>
<li>“<a href="https://www.researchgate.net/publication/304010295_Collective_Knowledge_Towards_RD_Sustainability">Collective Knowledge: towards R&D sustainability</a>”, Proceedings of the Conference on Design, Automation, and Test in Europe (DATE), 2016</li>
<li>“<a href="http://doi.acm.org/10.1145/2909437.2909449">Optimizing Convolutional Neural Networks on Embedded Platforms with OpenCL</a>”, IWOCL'16, Vienna, Austria, 2016</li>
<li><a href="https://scholar.google.fr/citations?view_op=view_citation&hl=en&user=IwcnpkwAAAAJ&cstart=20&citation_for_view=IwcnpkwAAAAJ:isC4tDSrTZIC">“Community-driven reviewing and validation of publications”</a>, Proceedings of the 1st ACM SIGPLAN Workshop on Reproducible Research Methodologies and New Publication Models in Computer Engineering @ PLDI’14, Edinburgh, UK</li>
</ol>
https://doi.org/10.5281/zenodo.3908799
oai:zenodo.org:3908799
eng
Zenodo
https://arxiv.org/abs/arXiv:1406.4020
https://doi.org/10.3850/9783981537079_1018
https://arxiv.org/abs/arXiv:1801.06378
https://doi.org/10.1007/s10766-010-0161-2
https://arxiv.org/abs/arXiv:1407.3487
https://zenodo.org/communities/artifact-evaluation
https://zenodo.org/communities/ck
https://doi.org/10.5281/zenodo.2544203
info:eu-repo/semantics/openAccess
Creative Commons Attribution 4.0 International
https://creativecommons.org/licenses/by/4.0/legalcode
reproducibility
artifact evaluation
open science
experiment automation
experiment crowdsourcing
research components
artifact sharing
artifact reuse
research artefacts
Enabling open and reproducible research at computer systems conferences: the good, the bad and the ugly
info:eu-repo/semantics/lecture
oai:zenodo.org:3726327
2020-06-22T14:09:56Z
user-mlperf
user-ck
Leo Gordon
Anton Lokhmotov
2020-03-24
<p><a href="https://developer.nvidia.com/tensorrt">TensorRT</a> plans generated on a <a href="https://developer.nvidia.com/embedded/jetson-agx-xavier-developer-kit">Jetson AGX Xavier Developer Kit</a> using code and instructions from <a href="https://github.com/mlperf/inference_results_v0.5/tree/master/closed/NVIDIA">NVIDIA's MLPerf Inference v0.5 submission</a> without any modifications.</p>
Also includes libnmsoptplugin.so
https://doi.org/10.5281/zenodo.3726327
oai:zenodo.org:3726327
Zenodo
https://zenodo.org/communities/mlperf
https://zenodo.org/communities/ck
https://doi.org/10.5281/zenodo.3726326
info:eu-repo/semantics/openAccess
Apache License 2.0
http://www.apache.org/licenses/LICENSE-2.0
MLPerf
Inference
TensorRT
Xavier
int8
original
MLPerf Inference v0.5 - TensorRT plans for NVIDIA Jetson AGX Xavier - int8, original
info:eu-repo/semantics/other
oai:zenodo.org:8144274
2023-07-14T02:26:44Z
openaire
user-cm
user-ck
Grigori Fursin
Arjun Suresh
2023-07-13
<p>This presentation is a part of the MLPerf inference submitter orientation. It explains how to make it easier to run MLPerf inference benchmarks out-of-the-box and automate submissions across diverse software, hardware, models and data from different vendors using the <a href="https://github.com/mlcommons/cm">MLCommons CM automation language</a> and <a href="https://access.cKnowledge.org">MLCommons CK playground</a>.</p>
<p>The documentation to run MLPerf benchmarks using CM is available <a href="https://github.com/mlcommons/ck/tree/master/docs/mlperf">here</a>.</p>
<p> </p>
https://doi.org/10.5281/zenodo.8144274
oai:zenodo.org:8144274
Zenodo
https://zenodo.org/communities/ck
https://zenodo.org/communities/cm
https://doi.org/10.5281/zenodo.8144273
info:eu-repo/semantics/openAccess
Creative Commons Attribution 4.0 International
https://creativecommons.org/licenses/by/4.0/legalcode
mlperf,automation,unification,portability,out-of-the-box,inference,submission
MLPerf inference submitter orientation: "making it easier to run MLPerf benchmarks out-of-the-box and submit results"
info:eu-repo/semantics/lecture
oai:zenodo.org:10605079
2024-02-02T11:15:26Z
openaire
user-cm
user-ck
Grigori Fursin
Arjun Suresh
2024-02-01
<p>This presentation is a part of the MLPerf inference v4.0 submitter orientation v. It explains how to automate your submissions across different implementations, models, data sets, frameworks, software and hardware from different vendors using <a href="https://github.com/mlcommons/ck/blob/master/docs/list_of_scripts.md">portable and technology-agnostic automation recipes</a> being developed by the <a href="https://github.com/mlcommons/ck/blob/master/docs/taskforce.md">MLCommons Task Force on Automation and Reproducibility</a>. You can find the documentation to run MLPerf benchmarks using CM <a href="https://github.com/mlcommons/ck/tree/master/docs/mlperf">here</a>. You can find current CM coverage for MLPerf inference <a href="https://github.com/mlcommons/ck/issues">here</a>.</p>
<p><em><a href="https://github.com/mlcommons/ck">MLCommons Collective Mind (CM)</a> is a community project based on your feedback - you can contact us via public Discord server if you have questions or suggestions!</em></p>
https://doi.org/10.5281/zenodo.10605079
oai:zenodo.org:10605079
Zenodo
https://zenodo.org/communities/ck
https://zenodo.org/communities/cm
https://doi.org/10.5281/zenodo.8144273
info:eu-repo/semantics/openAccess
Creative Commons Attribution 4.0 International
https://creativecommons.org/licenses/by/4.0/legalcode
mlperf,automation,unification,portability,out-of-the-box,inference,submission
MLPerf inference submitter orientation: "how to automate and reproduce your submissions"
info:eu-repo/semantics/lecture
oai:zenodo.org:2280136
2019-06-22T09:57:44Z
user-ck
The TensorFlow Authors
2018-12-14
<p>MobileNet models for <a href="https://www.tensorflow.org">TensorFlow </a>archived from <a href="https://github.com/tensorflow/models/blob/master/research/slim/nets/mobilenet_v1.md">github.com/tensorflow/models/blob/master/research/slim/nets/mobilenet_v1.md</a> to ensure the stability of <a href="https://github.com/ctuning/ck">Collective Knowledge</a> <a href="https://ReproIndex.com/components/&c=package">packages</a> and <a href="https://github.com/ctuning/ck-mlperf">workflows</a> used to automate the <a href="https://mlperf.org">MLPerf benchmark</a>.</p>
<p><a href="https://github.com/tensorflow/models/blob/master/LICENSE">Copyright The TensorFlow Authors. Apache License Version 2.0</a><br>
</p>
https://doi.org/10.5281/zenodo.2280136
oai:zenodo.org:2280136
Zenodo
https://arxiv.org/abs/1704.04861
https://zenodo.org/communities/ck
https://doi.org/10.5281/zenodo.2280135
info:eu-repo/semantics/openAccess
Apache License 2.0
http://www.apache.org/licenses/LICENSE-2.0
MLPerf
Model
inference
MobileNets
TensorFlow
v1
MobileNets-v1-20170614 for MLPerf Inference (Edge)
info:eu-repo/semantics/other
oai:zenodo.org:2459444
2020-01-25T07:24:11Z
software
user-ck
Leo Gordon
Flavio Vella
Grigori Fursin
Anton Lokhmotov
2018-12-20
<p>Collective Knowledge framework (CK) helps to convert ad-hoc code, data and scripts into portable, customizable and reusable components with a simple Python API and an integrated package manager for Linux, MacOS, Windows and Android; assemble automated workflows; crowdsource complex experiments; generate interactive papers, etc:</p>
<ul>
<li><a href="https://github.com/ctuning/ck">https://github.com/ctuning/ck</a></li>
<li><a href="http://cKnowledge.org">http://cKnowledge.org</a></li>
<li><a href="http://cKnowledge.org/partners">http://cKnowledge.org/partners</a></li>
<li><a href="http://cKnowledge.org/dashboard">http://cKnowledge.org/dashboard</a></li>
</ul>
<p>Example of reproducible and interactive article automatically generated by CK: <a href="http://cKnowledge.org/rpi-crowd-tuning">http://cKnowledge.org/rpi-crowd-tuning</a> .</p>
<p> </p>
https://doi.org/10.5281/zenodo.2459444
oai:zenodo.org:2459444
Zenodo
isbn:978-3-9815370-6-2
https://zenodo.org/communities/ck
https://doi.org/10.5281/zenodo.2459443
info:eu-repo/semantics/openAccess
BSD 3-Clause "New" or "Revised" License
https://opensource.org/licenses/BSD-3-Clause
Collective Knowledge
Collaborative Research
Reproducible Research
Open Science
Automated workflows
Customizable workflows
Portable workflows
Reusable artifacts
Python API
JSON API
experiment crowdsourcing
package manager
reproducible articles
interactive articles
artifact evaluation
Collective Knowledge Framework
info:eu-repo/semantics/other
oai:zenodo.org:2556147
2020-01-20T17:30:47Z
openaire
user-ck
Grigori Fursin
2019-02-03
<p>Validating experimental results from articles has finally become a norm at many HPC and systems conferences. Nowadays, more than half of accepted papers pass artifact evaluation and share related code and data. Unfortunately, lack of a common experimental framework, common research methodology and common formats places an increasing burden on evaluators to validate a growing number of ad-hoc artifacts. Furthermore, having too many ad-hoc artifacts and Docker snapshots is almost as bad as not having any (!), since they cannot be easily reused, customized and built upon.</p>
<p>While overviewing more than 100 papers during artifact evaluation at HPC conferences, we noticed that many of them use similar experimental setups, benchmarks, models, data sets, environments and platforms. This motivated us to develop Collective Knowledge (CK), an open workflow framework with a unified Python API to automate common researchers’ tasks such as detecting software and hardware dependencies, installing missing packages, downloading data sets and models, compiling and running programs, performing autotuning and co-design, crowdsourcing time-consuming experiments across computing resources provided by volunteers similar to SETI@home, reproducing results, automatically generating interactive articles, and so on: http://cKnowledge.org .</p>
<p>In this talk I will introduce CK concepts and present several real world use cases from the Raspberry Pi foundation, ACM, General Motors, Amazon and Arm on collaborative benchmarking, autotuning and co-design of efficient software/hardware stacks for emerging workloads including deep learning. I will also present our latest initiative to create an open repository of reusable research components and workflows at HPC conferences. We plan to use it to automate the Student Cluster Competition Reproducibility Challenge at the Supercomputing conference.</p>
Presentation at FOSDEM'19 (HPC, Big Data and Data Science): https://fosdem.org/2019/schedule/event/collective_knowledge
https://doi.org/10.5281/zenodo.2556147
oai:zenodo.org:2556147
eng
Zenodo
https://zenodo.org/communities/ck
https://doi.org/10.5281/zenodo.2556146
info:eu-repo/semantics/openAccess
Creative Commons Attribution 4.0 International
https://creativecommons.org/licenses/by/4.0/legalcode
experiment automation
collaborative research
reproducible research
open science
Collective Knowledge
crowdsource experiments
research API
adaptive workflows
portable workflows
Collective Knowledge (CK): an open-source framework to automate, reproduce, and crowdsource HPC experiments
info:eu-repo/semantics/lecture
oai:zenodo.org:4005773
2020-11-23T22:43:52Z
openaire
user-ck
Grigori Fursin
2020-08-28
<p><strong>Invited talk at FastPath 2020 (International Workshop on Performance Analysis of Machine Learning Systems) co-located with ISPASS 2020.</strong></p>
<ul>
<li><strong>Article: </strong><a href="https://arxiv.org/pdf/2011.01149.pdf">arxiv.org/pdf/2011.01149.pdf</a> ( <strong><a href="https://github.com/ctuning/ck">code</a> </strong>and<strong> <a href="https://github.com/ctuning/ai">data</a></strong> )</li>
<li><strong>Reproducibility initiative: </strong><a href="https://cTuning.org/ae">systems and ML conferences</a> ( <a href="https://cKnowledge.io/reproduced-papers">reproduced papers</a> and <a href="https://cKnowledge.io/reproduced-results">results</a> )</li>
<li><strong>Workshop program: </strong><a href="https://fastpath2020.github.io/Program">fastpath2020.github.io/Program</a></li>
<li><strong>Author: </strong><a href="https://cKnowledge.io/@gfursin">Grigori Fursin</a></li>
</ul>
<p><strong>Abstract:</strong></p>
<p>10 years ago we released our ML-based MILEPOST compiler with all related code and experimental data at <a href="https://ctuning.org/wiki/index.php/CDatabase">cTuning.org</a>. Unfortunately, this research quickly stalled after we struggled to reproduce performance results and predictive models shared by volunteers across rapidly changing systems.</p>
<p>In this talk, I will describe my 10-year effort to solve numerous reproducibility issues in ML&systems research. I will share my experience reproducing 150+ systems and ML papers during artifact evaluation at ASPLOS, MLSys, CGO, PPoPP and Supercomputing. This tedious experience motivated me to develop the <a href="https://github.com/ctuning/ck">Collective Knowledge framework</a> and the open <a href="https://cKnowledge.io">cKnowledge.io</a> portal to bring DevOps principles to our research. I will also present <a href="https://cKnowledge.io/solutions">cKnowledge solutions</a> - a new way to package and share research artifacts and results with common Python APIs, CLI actions, portable workflows and JSON meta descriptions. Such solutions can be used to automatically build, benchmark and validate ML&system experiments across continuously evolving platforms.</p>
<p>I will conclude with <a href="https://cKnowledge.org/partners">several practical use-cases</a> of our technology in collaboration with Arm, IBM, General Motors, the Raspberry Pi foundation and MLPerf. Our long-term goal is to help researchers share their new ML techniques as production-ready packages along with published papers and participate in collaborative and reproducible benchmarking, co-design and comparison of efficient ML/software/hardware stacks.</p>
<p> </p>
https://doi.org/10.5281/zenodo.4005773
oai:zenodo.org:4005773
eng
Zenodo
https://zenodo.org/communities/ck
https://doi.org/10.5281/zenodo.4005587
info:eu-repo/semantics/openAccess
Creative Commons Attribution 4.0 International
https://creativecommons.org/licenses/by/4.0/legalcode
reproducibility
reusability
portability
workflow
automation
machine learning
systems
benchmarking
artifact evaluation
fair principles
open science
Enabling reproducible ML and Systems research: the good, the bad, and the ugly
info:eu-repo/semantics/lecture
oai:zenodo.org:4005600
2020-08-29T00:59:24Z
openaire
user-ck
Grigori Fursin
2020-08-28
<p><strong>Invited talk at Google compiler+ML seminar</strong></p>
<p>I was asked to share my experience with the <a href="https://en.wikipedia.org/wiki/MILEPOST_GCC">MILEPOST project</a> to build ML-based self-optimizing compiler, <a href="https://cTuning.org">cTuning.org</a> to crowdsource optimization, reproducibility issues, and my future directions.</p>
<p> </p>
https://doi.org/10.5281/zenodo.4005600
oai:zenodo.org:4005600
eng
Zenodo
https://zenodo.org/communities/ck
https://doi.org/10.5281/zenodo.4005599
info:eu-repo/semantics/openAccess
Creative Commons Attribution 4.0 International
https://creativecommons.org/licenses/by/4.0/legalcode
milepost project
machine learning
compilers
autotuning
collaboration
reproducibility
MILEPOST Project Experience: building an ML-based self-optimizing compiler
info:eu-repo/semantics/lecture
oai:zenodo.org:2544204
2020-06-26T03:59:31Z
user-artifact-evaluation
openaire
user-ck
Grigori Fursin
2017-03-14
<p><em>14 March 2017, CNRS webinar, Grenoble, France (original slides were shared <a href="https://www.slideshare.net/GrigoriFursin/enabling-open-and-reproducible-computer-systems-research-the-good-the-bad-and-the-ugly">here</a>).</em></p>
<p>A decade ago <a href="http://fursin.net/research.html">my research</a> nearly stalled. I was investigating how to crowdsource performance analysis and optimization of realistic workloads across diverse hardware provided by volunteers and combine it with machine learning [1]. Often, it was simply impossible to reproduce crowdsourced empirical results and build predictive models due to continuously changing software and hardware stacks. Worse still, lack of realistic workloads and representative data sets in our community severely limited the usefulness of such models.<br>
<br>
All these problems motivated me to create a public portal (<a href="http://cTuning.org">cTuning.org</a>) to share, validate and reuse workloads, data sets, tools, experimental results, and predictive models while involving the community in this effort [2]. This experience, in turn, helped us to initiate the so-called <a href="http://ctuning.org/ae">Artifact Evaluation</a> (AE) at ACM conferences on parallel programming, architecture and code generation (ASPLOS, CGO, PPoPP, PACT, SC and MLSys). AE aims to independently validate experimental results reported in the publications and to encourage code and data sharing.</p>
<p>These slides are from my webinar <a href="https://github.com/alegrand/RR_webinars/blob/master/8_artifact_evaluation/index.org"><em>“Enabling open and reproducible research at computer systems conferences: the good, the bad and the ugly”</em></a> at CNRS Grenoble (14 March 2017). I shared my practical experience organizing Artifact Evaluation over the past years, along with encountered problems and possible solutions.<br>
<br>
On the one hand, we have received incredible support from the research community, <a href="http://acm.org/">ACM</a>, universities, and companies. We have even received a record number of artifact submissions at the <a href="http://ctuning.org/ae">CGO/PPoPP'17</a> AE (27 vs 17 two years ago) sponsored by <a href="http://research.nvidia.com/">NVIDIA</a> and the <a href="http://ctuning.org/">cTuning foundation</a>. We have also introduced <a href="https://github.com/ctuning/ck-artifact-evaluation/blob/master/wfe/artifact-evaluation/templates/ae-20190108.tex">Artifact Appendices</a> and co-authored the new <a href="http://www.acm.org/publications/policies/artifact-review-badging">ACM Result and Artifact Review and Badging</a> policy now used at <a href="http://sc17.supercomputing.org/submitters/technical-papers/reproducibility-initiatives-for-technical-papers">Supercomputing</a>. <br>
<br>
On the other hand, the use of proprietary benchmarks, rare hardware platforms, and totally ad-hoc scripts to set up, run and process experiments all place a huge burden on evaluators. It is simply too difficult and time-consuming to customize and rebuild experimental setups, reuse artifacts and eventually build upon others’ efforts - the main pillars of open science!<br>
<br>
I then present <a href="http://cknowledge.org/">Collective Knowledge (CK)</a>, my attempt to introduce a customizable workflow framework with a unified JSON API and a <a href="https://github.com/ctuning/ck/wiki/Portable-workflows">cross-platform package manager</a>, that can automate ML&systems R&D and enable live papers while automatically adapting to continuously evolving software and hardware [3]. I also demonstrate a practical CK workflow to <a href="http://cknowledge.org/ai">collaboratively optimize deep learning</a> across different compilers, libraries, data sets and diverse platforms from resource-constrained mobile devices to data centers (see our <a href="http://cknowledge.org/android-apps.html">Android app to crowdsource DNN optimization</a> across diverse mobile devices provided by volunteers, and the <a href="http://cknowledge.org/repo-beta">public repository with results</a>) [4].<br>
<br>
Finally, I describe our novel publication model to reproduce results from published papers with the help of the community [5]. <br>
<br>
<em>Please feel free to contact me at <a href="mailto:Grigori.Fursin@cTuning.org">Grigori.Fursin@cTuning.org</a> if you have any questions or comments! I am looking forward to your feedback! </em></p>
<p><strong>References</strong></p>
<ol>
<li><a href="https://scholar.google.com/citations?view_op=view_citation&hl=en&user=IwcnpkwAAAAJ&citation_for_view=IwcnpkwAAAAJ:LkGwnXOMwfcC">“Milepost GCC: Machine learning enabled self-tuning compiler”, International journal of parallel programming</a>, Volume 39, Issue 3, pp.296-327, 2009</li>
<li><a href="https://hal.inria.fr/inria-00436029v2">“Collective Tuning Initiative: automating and accelerating development and optimization of computing systems”</a>, GCC Developers' Summit, Montreal, Canada. 2009</li>
<li>“<a href="https://www.researchgate.net/publication/304010295_Collective_Knowledge_Towards_RD_Sustainability">Collective Knowledge: towards R&D sustainability</a>”, Proceedings of the Conference on Design, Automation, and Test in Europe (DATE), 2016</li>
<li>“<a href="http://doi.acm.org/10.1145/2909437.2909449">Optimizing Convolutional Neural Networks on Embedded Platforms with OpenCL</a>”, IWOCL'16, Vienna, Austria, 2016</li>
<li><a href="https://scholar.google.fr/citations?view_op=view_citation&hl=en&user=IwcnpkwAAAAJ&cstart=20&citation_for_view=IwcnpkwAAAAJ:isC4tDSrTZIC">“Community-driven reviewing and validation of publications”</a>, Proceedings of the 1st ACM SIGPLAN Workshop on Reproducible Research Methodologies and New Publication Models in Computer Engineering @ PLDI’14, Edinburgh, UK</li>
</ol>
https://doi.org/10.5281/zenodo.2544204
oai:zenodo.org:2544204
eng
Zenodo
https://arxiv.org/abs/arXiv:1406.4020
https://doi.org/10.3850/9783981537079_1018
https://arxiv.org/abs/arXiv:1801.06378
https://doi.org/10.1007/s10766-010-0161-2
https://arxiv.org/abs/arXiv:1407.3487
https://zenodo.org/communities/artifact-evaluation
https://zenodo.org/communities/ck
https://doi.org/10.5281/zenodo.2544203
info:eu-repo/semantics/openAccess
Creative Commons Attribution 4.0 International
https://creativecommons.org/licenses/by/4.0/legalcode
reproducibility
artifact evaluation
open science
experiment automation
experiment crowdsourcing
research components
artifact sharing
artifact reuse
research artefacts
Enabling open and reproducible research at computer systems conferences: the good, the bad and the ugly
info:eu-repo/semantics/lecture
oai:zenodo.org:7143424
2022-10-04T14:26:21Z
openaire
user-cm
user-ck
Grigori Fursin
2022-10-04
<p>The 1st presentation to help prepare a <a href="https://github.com/mlcommons/ck/blob/master/docs/mlperf-education-workgroup.md">new MLCommons workgroup</a> to make it easier to run, customize and reproduce MLPerf benchmarks.</p>
<p>The mission:</p>
<ul>
<li>Develop an automated open-source workflow to make it easier to plug any real-world ML & AI tasks, models, data sets, software and hardware into the MLPerf benchmarking infrastructure.</li>
<li>Use this workflow to help the newcomers learn how to customize and run MLPerf benchmarks across rapidly evolving software, hardware and data.</li>
<li>Lower the barrier of entry for new MLPerf submitters and reduce their associated costs.</li>
<li>Automate design space exploration of diverse ML/SW/HW stacks to trade off performance, accuracy, energy, size and costs; automate submission of Pareto-efficient configurations to MLPerf.</li>
<li>Help end-users visualize all MLPerf results, reproduce them and deploy the most suitable ML/SW/HW stacks in production.</li>
<li>Support reproducibility initiatives at ML and Systems conferences using rigorous MLPerf methodology and our educational toolkit.</li>
</ul>
<p> </p>
https://doi.org/10.5281/zenodo.7143424
oai:zenodo.org:7143424
Zenodo
https://zenodo.org/communities/ck
https://zenodo.org/communities/cm
https://doi.org/10.5281/zenodo.7143423
info:eu-repo/semantics/openAccess
Creative Commons Attribution 4.0 International
https://creativecommons.org/licenses/by/4.0/legalcode
mlperf
benchmark
inference
machine learning
artificial intelligence
collective knowledge
collective mind
workflow automation
design space exploration
reproducibility
deployment
Preparing a new MLCommons education and reproducibility workgroup to make it easier to run, customize and reproduce MLPerf benchmarks
info:eu-repo/semantics/lecture
oai:zenodo.org:3361502
2019-08-06T16:41:08Z
user-mlperf
user-ck
The TensorFlow Authors
dividiti
2019-08-06
<p>SSD-MobileNet-v1 models used in MLPerf Inference:</p>
<ul>
<li>A TensorFlow model archived from the <a href="https://github.com/tensorflow/models/blob/master/research/object_detection/README.md">TensorFlow Object Detection model zoo</a>.</li>
<li>A TFLite model obtained by <a href="http://dividiti.com">dividiti</a> from the above by using <a href="https://github.com/ctuning/ck-mlperf/tree/master/package/model-tflite-mlperf-ssd-mobilenet">instructions</a> adapted from <a href="https://medium.com/tensorflow/training-and-serving-a-realtime-mobile-object-detector-in-30-minutes-with-cloud-tpus-b78971cf1193">Google's blog</a>.</li>
</ul>
<p> </p>
https://doi.org/10.5281/zenodo.3361502
oai:zenodo.org:3361502
Zenodo
https://zenodo.org/communities/mlperf
https://zenodo.org/communities/ck
https://doi.org/10.5281/zenodo.3361501
info:eu-repo/semantics/openAccess
Apache License 2.0
http://www.apache.org/licenses/LICENSE-2.0
MLPerf
Inference
SSD-MobileNet-v1
TensorFlow
TFLite
TF/TFLite SSD-MobileNet models (used in MLPerf Inference)
info:eu-repo/semantics/other
oai:zenodo.org:2269307
2019-04-10T02:10:58Z
user-ck
The TensorFlow Authors
2018-12-14
<p>MobileNet models for <a href="https://www.tensorflow.org">TensorFlow </a>archived from <a href="https://github.com/tensorflow/models/blob/master/research/slim/nets/mobilenet_v1.md">github.com/tensorflow/models/blob/master/research/slim/nets/mobilenet_v1.md</a> to ensure the stability of Collective Knowledge <a href="http://cKnowledge.org/shared-packages.html">packages</a> and <a href="https://github.com/ctuning/ck-mlperf">workflows</a> used to automate the <a href="https://mlperf.org">MLPerf benchmark</a>.</p>
<p><a href="https://github.com/tensorflow/models/blob/master/LICENSE">Copyright The TensorFlow Authors. Apache License Version 2.0</a><br>
</p>
https://doi.org/10.5281/zenodo.2269307
oai:zenodo.org:2269307
Zenodo
https://arxiv.org/abs/1704.04861
https://zenodo.org/communities/ck
https://doi.org/10.5281/zenodo.2269306
info:eu-repo/semantics/openAccess
Apache License 2.0
http://www.apache.org/licenses/LICENSE-2.0
MLPerf
Model
inference
MobileNets
TensorFlow
v1
MobileNets-v1-20180802 for MLPerf Inference (Edge)
info:eu-repo/semantics/other
oai:zenodo.org:7871070
2023-04-27T14:26:42Z
user-artifact-evaluation
openaire
user-cm
user-ck
Grigori Fursin
Arjun Suresh
2023-04-27
<p>This presentation introduces <a href="https://github.com/mlcommons/ck/tree/master/platform">Collective Knowledge Playground</a> - a free, open-source and technology-agnostic on-prem platform being developed by the <a href="https://cknowledge.org/mlcommons-taskforce">MLCommons taskforce on automation and reproducibility</a>. </p>
<p>Our goal is to let the community benchmark, optimize and compare AI, ML and other emerging applications across diverse and rapidly evolving models, software, hardware and data from different vendors in terms of costs, performance, power consumption, accuracy, size and other metrics in a unified, collaborative, automated and reproducible way.</p>
<p>This platform is powered by the portable and technology-agnostic <a href="https://github.com/mlcommons/ck">MLCommons Collective Mind automation framework (CM aka CK2)</a> with <a href="https://github.com/mlcommons/ck/tree/master/cm-mlops/script">portable and reusable automation recipes</a> developed by the community to solve the "AI/ML dependency hell" and automatically connect diverse and continuously changing models, software, hardware, data sets, best practices and optimization techniques into end-to-end applications in a transparent and non-intrusive way.</p>
<p>Our vision for the CK platform is to help researchers, engineers and entrepreneurs accelerate innovation by automatically generating the most efficient, reproducible and deployable full-stack AI/ML applications using the most suitable software/hardware stack at any given time (model, framework, inference engine and any other related dependency) based on their requirements and constraints including costs, throughput, latency, power consumption, accuracy, target devices (cloud/edge/mobile/tiny), environment and data while slashing their research, development and operational costs.</p>
<p>See this <a href="https://www.youtube.com/watch?v=7zpeIVwICa4">ACM tech talk</a> and <a href="https://arxiv.org/abs/2011.01149">journal article</a> to learn more about our motivation.</p>
https://doi.org/10.5281/zenodo.7871070
oai:zenodo.org:7871070
eng
Zenodo
https://zenodo.org/communities/artifact-evaluation
https://zenodo.org/communities/ck
https://zenodo.org/communities/cm
https://doi.org/10.5281/zenodo.7871069
info:eu-repo/semantics/openAccess
Creative Commons Attribution 4.0 International
https://creativecommons.org/licenses/by/4.0/legalcode
machine learning systems
machine learning
artificial intelligence
automation
benchmarking
optimization
co-design
reproducibility
reusability
workflow
collective knowledge
collective mind
ctuning
ck playground
mlperf
mlcommons
Collective Knowledge Playground
info:eu-repo/semantics/lecture
oai:zenodo.org:2544258
2020-01-20T15:21:28Z
openaire
user-ck
Grigori Fursin
2017-09-14
<p>The original presentation was shared via <a href="https://www.slideshare.net/GrigoriFursin/adapting-to-a-cambrian-aiswhw-explosion-with-open-codesign-competitions-and-collective-knowledge">SlideShare</a>.</p>
<p>Slides from the ARM's Research Summit'17 about the "Community-Driven and Knowledge-Guided Optimization of AI Applications Across the Whole SW/HW Stack":</p>
<ul>
<li><a href="http://cKnowledge.org">cKnowledge.org</a></li>
<li><a href="http://cKnowledge.org/repo">cKnowledge.org/repo</a></li>
<li><a href="http://cKnowledge.org/repo-beta">cKnowledge.org/repo-beta</a></li>
<li><a href="http://cknowledge.org/android-apps.html">cKnowledge.org/android-apps.html</a></li>
<li><a href="http://cKnowledge.org/ai">cKnowledge.org/ai</a></li>
<li><a href="https://developer.arm.com/research/summit">developer.arm.com/research/summit</a></li>
</ul>
<p>Co-designing the whole AI/SW/HW stack in terms of speed, accuracy, energy consumption, size, costs, and other metrics has become extremely complex, long and costly. With no rigorous methodology for analyzing performance and accumulating optimisation knowledge, we are simply destined to drown in the ever growing number of design choices, system<br>
features and conflicting optimisation goals.<br>
<br>
We present our novel community-driven approach to solve the above problems. Originating from natural sciences, this approach is embodied in Collective Knowledge (CK), our open-source cross-platform workflow framework and repository for automatic, collaborative and reproducible experimentation. CK helps organize, unify and share representative workloads, data sets, AI frameworks, libraries, compilers, scripts, models and other artifacts as customizable and reusable components with a common JSON API.<br>
<br>
CK helps bring academia, industry and end-users together to gradually expose optimisation choices at all levels (e.g. from parameterized models and algorithmic skeletons to compiler flags and hardware configurations) and autotune them across diverse inputs and platforms. Optimization knowledge gets continuously aggregated in public or private repositories such as cKnowledge.org/repo in a reproducible way, and can be then mined and extrapolated to predict better AI algorithm choices, compiler transformations and hardware designs.<br>
<br>
We also demonstrate how we use this approach in practice together with ARM and other companies to adapt to a Cambrian AI/SW/HW explosion by creating an open repository of reusable AI artifacts, and then collaboratively optimising and co-designing the whole deep learning stack (software, hardware and models).</p>
https://doi.org/10.5281/zenodo.2544258
oai:zenodo.org:2544258
eng
Zenodo
https://doi.org/10.3233/SPR-140396
https://arxiv.org/abs/arXiv:1506.06256
https://arxiv.org/abs/arXiv:1406.4020
https://doi.org/10.3850/9783981537079_1018
https://arxiv.org/abs/arXiv:1407.3487
https://zenodo.org/communities/ck
https://doi.org/10.5281/zenodo.2544257
info:eu-repo/semantics/openAccess
Creative Commons Attribution 4.0 International
https://creativecommons.org/licenses/by/4.0/legalcode
collaborative autotuning
sw/hw co-design
crowdsourcing co-design
crowdsourcing learning
self-optimizing systems
self-learning systems
repository of knowledge
open science
reproducibility
Adapting to a Cambrian AI/SW/HW explosion with open co-design competitions and Collective Knowledge
info:eu-repo/semantics/lecture
oai:zenodo.org:8105339
2023-11-05T14:59:11Z
openaire
user-cm
user-ck
Fursin, Grigori
2023-06-28
<p>The <a href="https://acm-rep.github.io/2023/keynotes">keynote presentation</a> from the <a href="https://acm-rep.github.io/2023/">1st ACM conference on reproducibility and replicability</a> (ACM REP'23).</p><p>The video of this presentation is available at the <a href="https://youtu.be/_1f9i_Bzjmg?si=vVWJGmqJQ3UverrN">ACM YouTube channel</a>.</p><p><i>Please don't hesitate to provide your feedback via the </i><a href="https://discord.gg/JjWNWXKxwT"><i>public Discord server</i></a><i> from the MLCommons Task Force on Automation and Reproducibility and </i><a href="https://github.com/mlcommons/ck/issues"><i>GitHub issues</i></a><i>.</i></p><p>[ <a href="https://github.com/mlcommons/ck">GitHub project</a> ] [ <a href="https://access.cKnowledge.org">Public Collective Knowledge repository</a> ]</p><p>[ <a href="https://cTuning.org/ae">Related reproducibility initiatives</a> ] [ <a href="https://cTuning.org">cTuning.org</a> ] [ <a href="https://cKnowledge.org">cKnowledge.org</a> ]</p><p>During the past 10 years, we have considerably improved the reproducibility of experimental results from published papers by introducing the artifact evaluation process with a unified artifact appendix and reproducibility checklists, Jupyter notebooks, containers, and Git repositories. On the other hand, our experience reproducing more than 200 papers shows that it can take weeks and months of painful and repetitive interactions between teams to reproduce artifacts. This effort includes decrypting numerous README files, examining ad-hoc artifacts and containers, and figuring out how to reproduce computational results. Furthermore, snapshot containers pose a challenge to optimize algorithms' performance, accuracy, power consumption and operational costs across diverse and rapidly evolving software, hardware, and data used in the real world.</p><p>In this talk, I explain how our practical artifact evaluation experience and the feedback from researchers and evaluators motivated us to develop a simple, intuitive, technology agnostic, and English-like scripting language called <a href="https://github.com/mlcommons/ck/blob/master/docs/README.md">Collective Mind (CM)</a>. It helps to automatically adapt any given experiment to any software, hardware, and data while automatically generating unified README files and synthesizing modular containers with a unified API. It is being developed by MLCommons to facilitate reproducible AI/ML Systems research and minimizing manual and repetitive benchmarking and optimization efforts, reduce time and costs for reproducible research, and simplify technology transfer to production. I also present several recent use cases of how CM helps <a href="https://cknowledge.org/mlperf-inf-v3.0-forbes">MLCommons</a>, the <a href="https://github.com/mlcommons/ck/blob/master/docs/tutorials/sc22-scc-mlperf.md">Student Cluster Competition</a>, and <a href="https://cTuning.org/ae">artifact evaluation</a> at ACM/IEEE conferences. I conclude with our development plans, new challenges, possible solutions, and upcoming reproducibility and optimization challenges powered by the MLCommons Collective Knowledge platform and CM: <a href="https://access.cknowledge.org/">access.cKnowledge.org</a>.</p>
https://doi.org/10.5281/zenodo.8105339
oai:zenodo.org:8105339
Zenodo
https://zenodo.org/communities/ck
https://zenodo.org/communities/cm
https://doi.org/10.5281/zenodo.8105338
info:eu-repo/semantics/openAccess
Creative Commons Attribution 4.0 International
https://creativecommons.org/licenses/by/4.0/legalcode
collective mind
collective knowledge
automation
reproducibility
replicability
reusability
performance
machine learning
artificial intelligence
systems
artifact evaluation
optimization challenges
competitions
mlcommons
mlperf
cTuning
cknowledge
chatgpt
llm
llm automation
Toward a common language to facilitate reproducible research and technology transfer: challenges and solutions
info:eu-repo/semantics/lecture