Important Dates
Paper decision: January 4, 2019
Artifact submission: January 18, 2019
Decision announced: February 22, 2019
Camera-ready paper: TBA
Conference: March 31 - April 2, 2019
Public discussion: TBA

SysML'19 AE Chairs

Gennady Pekhimenko (University of Toronto)
Grigori Fursin (cTuning foundation / dividiti)

SysML'19 AE Committee

TBA

Artifact Evaluation for SysML 2019

[ Back to SysML 2019 conference website ]

SysML promotes reproducibility of experimental results and encourages code and data sharing to help the community quickly validate and compare alternative approaches. Authors of accepted SysML'19 papers are invited to formally describe supporting material (code, data, models, workflows, results) using the standard Artifact Appendix template and submit it to the Artifact Evaluation process (AE). Note that this submission is voluntary, will not influence the final decision regarding the papers, and promotes artifact sharing and reproducible research! For example, ACM Digital Library already allows one to find papers with available artifacts and reproducible results!

AE is run by a separate committee whose task is to assess how submitted artifacts support the work described in accepted papers based on the standard ACM Artifact Review and Badging policy. However, since it may be very time consuming to perform full validation of AI/ML experiments while requiring expensive computational resources, we decided to only validate if submitted artifacts are "available" and "functional" at SysML'19. Thus, depending on evaluation results, camera-ready papers will include artifact appendix and will receive at most two ACM stamps of approval printed on their first page:

   
We will discuss how to introduce the full validation of all experiments at future SysML with the community during public Artifact Evaluation discussion session at SysML'19.

You need to prepare your artifacts and appendix using the following guidelines. You can then submit your paper with artifact appendix via dedicated SysML AE website before January 18, 2019. Your submission will be then reviewed according to the following guidelines. Please, do not forget to provide a list of hardware, software, benchmark and data set dependencies in your artifact abstract - this is essential to find appropriate evaluators! At the end of the process we will inform you about how to add badges to your camera-ready paper.

If you plan to use Collective Knowledge framework to automate your workflow and require some assistance, please get in touch with the cTuning foundation and we will register you on the CK slack channel.

If you have questions, please check AE FAQs, discuss them via the dedicated AE google group or get in touch with SysML AE chairs!