There is a newer version of this record available.

Dataset Closed Access

Antarex HPC Fault Dataset

Alessio Netti; Zeynep Kiziltan; Ozalp Babaoglu; Alina Sirbu; Andrea Bartolini; Andrea Borghesi

Dublin Core Export

<?xml version='1.0' encoding='utf-8'?>
<oai_dc:dc xmlns:dc="" xmlns:oai_dc="" xmlns:xsi="" xsi:schemaLocation="">
  <dc:contributor>Alessio Netti</dc:contributor>
  <dc:contributor>Zeynep Kiziltan</dc:contributor>
  <dc:contributor>Ozalp Babaoglu</dc:contributor>
  <dc:contributor>Alina Sirbu</dc:contributor>
  <dc:contributor>Andrea Bartolini</dc:contributor>
  <dc:contributor>Andrea Borghesi</dc:contributor>
  <dc:creator>Alessio Netti</dc:creator>
  <dc:creator>Zeynep Kiziltan</dc:creator>
  <dc:creator>Ozalp Babaoglu</dc:creator>
  <dc:creator>Alina Sirbu</dc:creator>
  <dc:creator>Andrea Bartolini</dc:creator>
  <dc:creator>Andrea Borghesi</dc:creator>
  <dc:description>The Antarex dataset contains trace data collected from the homonymous experimental HPC system located at ETH Zurich while it was subjected to fault injection, for the purpose of conducting machine learning-based fault detection studies for HPC systems. Acquiring our own dataset was made necessary by the fact that commercial HPC system operators are very reluctant to share trace data containing information about faults in their systems.

In order to acquire data, we executed benchmark applications and at the same time injected faults in the system at specific times via dedicated programs, so as to trigger anomalies in the behaviour of the applications. A wide range of faults is covered in our dataset, from hardware faults, to misconfiguration faults, and finally to performance anomalies cause by interference from other processes. This was achieved through the FINJ fault injection tool, developed by the authors.

The dataset contains two types of data: one type of data refers to a series of CSV files, each containing a set of system performance metrics sampled through the LDMS HPC monitoring framework. Another type refers to the log files detailing the status of the system (i.e., currently running benchmark applications or injected fault programs) at each time point in the dataset. Such a structure enables researchers to perform a wide range of studies on the dataset. Moreover, since we collected the dataset by streaming continuous data, any study based on it will easily be reproducible on a real HPC system, in an online way. The dataset is divided in two parts: the first includes only the CPU and memory-related benchmark applications and fault programs, while the second is strictly hard drive-related. We executed each part in both single-core and multi-core variants, resulting in a total of 4 dataset blocks for 32 days of data acquisition, and 20GB of uncompressed data.

For a detailed analysis on the structure and features of the Antarex dataset, please refer to the research paper "Online Fault Classification in HPC System through Machine Learning", by Netti et al. Additional details can be found in the research paper "FINJ: a Fault Injection Tool for HPC System" by Netti et al., whereas all source code can be found on the GitHub repository of the FINJ tool.</dc:description>
  <dc:description>The archive contains 4 directories, one for each block of the dataset - namely CPU/Memory and HDD, in single-core and multi-core variants. In each of these directories, you will find the following: a 7z archive containing the LDMS CSV files for each of the 7 used plugins; FINJ workloads and execution logs; the histograms for the durations and inter-arrival times of fault tasks in PDF format; launch scripts, if any. Source code for all of the injected fault programs and additional details can be found on the GitHub repository of the FINJ tool.</dc:description>
  <dc:subject>High-performance computing</dc:subject>
  <dc:subject>Exascale systems</dc:subject>
  <dc:subject>Fault Detection</dc:subject>
  <dc:subject>Machine Learning</dc:subject>
  <dc:title>Antarex HPC Fault Dataset</dc:title>
All versions This version
Views 73433
Downloads 2763
Data volume 367.3 GB1.6 GB
Unique views 62029
Unique downloads 1782


Cite as