CHESS: A Framework for Evaluation of Self-adaptive Systems based on Chaos Engineering

This repository contains the VM for CHESS: A Framework for Evaluation of Self-adaptive Systems based on Chaos Engineering

This artifact was used in the paper "On Evaluating Self-Adaptive and Self-Healing Systems using Chaos Engineering", published in the 3rd IEEE International Conference on Autonomic Computing and Self-Organizing Systems (ACSOS), 2022. A preprint of the paper is included in this repository.  A paper further detailing the artifact is under submission for the artifact track at the 18th Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS) and will be added in due time.

A demo video of using the artifact is available on YouTube: https://youtu.be/CBcaPJgpi-o

Abstract—With the growing adoption of self-adaptive systems in various domains, there is an increasing need for strategies to assess their correct behavior. In particular self-healing systems, which aim to provide resilience and fault-tolerance, often deal with unanticipated failures in critical and highly dynamic environments. Their reactive and complex behavior makes it challenging to assess if these systems execute according to the desired goals. Recently, several studies have expressed concern about the lack of systematic evaluation methods for self-healing behavior.

In the paper, we propose CHESS, an approach for the systematic evaluation of self-adaptive and self-healing systems that builds on chaos engineering. Chaos engineering is a methodology for subjecting a system to unexpected conditions and scenarios. It has shown great promise in helping developers build resilient micro-service architectures and cyber-physical systems. CHESS turns this idea around by using chaos engineering to evaluate how well a self-healing system can withstand such perturbations. We investigate the viability of this approach through a case study that runs chaos experiments on a self-healing smart office environment. These experiments help us explore the promises and limitations of the approach, as well as identify directions where additional work is needed. We conclude with the lessons learned while evaluating self-healing systems at the infrastructural and functional level.

Artifact

The artifact provided here is a VMware image for a 64-bit Ubuntu 20.04LTS VM containing both the self-healing smart office environment (as a Kubernetes cluster) and the CHESS infrastructure used to conduct the self-healing and self-adaptation evaluations presented in the paper.  There are detailed READMEs in the VM that discuss the experiments.

Citation

If you build on this artifact, please cite our work by referring to the paper:

@inproceedings{naqvi2022:evaluating,
  title = {On Evaluating Self-Adaptive and Self-Healing Systems Using Chaos Engineering},
  booktitle = {IEEE International Conference on Autonomic Computing and Self-Organizing Systems (ACSOS)},
  author = {Naqvi, Moeen Ali and Malik, Sehrish and Astekin, Merve and Moonen, Leon},
  year = {2022},
  month = sep,
  publisher = {IEEE}
  keywords = {self-healing, resilience, chaos engineering, evaluation, exploratory study}
}