Dataset Open Access

Multi-Source Distributed System Data for AI-powered Analytics

Sasho Nedelkoski; Jasmin Bogatinovski; Ajay Kumar Mandapati; Soeren Becker; Jorge Cardoso; Odej Kao


JSON-LD (schema.org) Export

{
  "inLanguage": {
    "alternateName": "eng", 
    "@type": "Language", 
    "name": "English"
  }, 
  "description": "<p><strong>Abstract:</strong></p>\n\n<p>In recent years there has been an increased interest in Artificial Intelligence for IT Operations (AIOps). This field utilizes monitoring data from IT systems, big data platforms, and machine learning to automate various operations and maintenance (O&amp;M) tasks for distributed systems.<br>\nThe major contributions have been materialized in the form of novel algorithms.<br>\nTypically, researchers took the challenge of exploring one specific type of observability data sources, such as application logs, metrics, and distributed traces, to create new algorithms.<br>\nNonetheless, due to the low signal-to-noise ratio of monitoring data, there is a consensus that only the analysis of multi-source monitoring data will enable the development of useful algorithms that have better performance. &nbsp;<br>\nUnfortunately, existing datasets usually contain only a single source of data, often logs or metrics. This limits the possibilities for greater advances in AIOps research.<br>\nThus, we generated high-quality multi-source data composed of distributed traces, application logs, and metrics from a complex distributed system. This paper provides detailed descriptions of the experiment, statistics of the data, and identifies how such data can be analyzed to support O&amp;M tasks such as anomaly detection, root cause analysis, and remediation.</p>\n\n<p><strong>General Information:</strong></p>\n\n<p>This repository contains the simple scripts for data statistics, and link to the multi-source distributed system dataset.</p>\n\n<p>You may find details of this dataset from the original paper:</p>\n\n<p><em>Sasho Nedelkoski, Jasmin Bogatinovski, Ajay Kumar Mandapati, Soeren Becker, Jorge Cardoso, Odej Kao, &quot;Multi-Source Distributed System Data for AI-powered Analytics&quot;.&nbsp;</em></p>\n\n<p><strong>If you use the data, implementation, or any details of the paper, please cite!</strong></p>\n\n<p>&nbsp;</p>\n\n<p>BIBTEX:</p>\n\n<p>_________________________________________</p>\n\n<pre>@inproceedings{nedelkoski2020multi,\n  title={Multi-source Distributed System Data for AI-Powered Analytics},\n  author={Nedelkoski, Sasho and Bogatinovski, Jasmin and Mandapati, Ajay Kumar and Becker, Soeren and Cardoso, Jorge and Kao, Odej},\n  booktitle={European Conference on Service-Oriented and Cloud Computing},\n  pages={161--176},\n  year={2020},\n  organization={Springer}\n}\n</pre>\n\n<p>___________________________</p>\n\n<p>The multi-source/multimodal dataset is composed of distributed traces, application logs, and metrics produced from running a complex distributed system (Openstack). In addition, we also provide the workload and fault scripts together with the Rally report which can serve as ground truth. We provide two datasets, which differ on how the workload is executed. The&nbsp;<em><strong>sequential_data</strong>&nbsp;</em>is generated via executing workload of sequential user requests. The <strong><em>concurrent_data&nbsp;</em></strong>is generated via executing workload of concurrent user requests.</p>\n\n<p>The raw logs in both datasets contain the same files. If the user wants the logs filetered by time with respect to the two datasets, should refer to the timestamps at the metrics (they provide the time window).&nbsp;<strong>In addition, we suggest to use the provided aggregated time ranged logs for both datasets in CSV format.</strong></p>\n\n<p><strong><strong>Important:</strong>&nbsp;The logs and the metrics are synchronized with respect time and they are both recorded on CEST (central european standard time). The traces are on UTC (Coordinated Universal Time -2 hours). They should be synchronized if the user develops multimodal methods. Please read the IMPORTANT_experiment_start_end.txt file before working with the data.</strong></p>\n\n<p>Our GitHub repository with the code for the workloads and scripts for basic analysis can be found at:&nbsp;<a href=\"https://github.com/SashoNedelkoski/multi-source-observability-dataset/\">https://github.com/SashoNedelkoski/multi-source-observability-dataset/</a></p>", 
  "license": "https://creativecommons.org/licenses/by/4.0/legalcode", 
  "creator": [
    {
      "affiliation": "Complex and Distributed IT-Systems Group, TU Berlin, Berlin, Germany", 
      "@type": "Person", 
      "name": "Sasho Nedelkoski"
    }, 
    {
      "affiliation": "Complex and Distributed IT-Systems Group, TU Berlin, Berlin, Germany", 
      "@type": "Person", 
      "name": "Jasmin Bogatinovski"
    }, 
    {
      "affiliation": "Complex and Distributed IT-Systems Group, TU Berlin, Berlin, Germany", 
      "@type": "Person", 
      "name": "Ajay Kumar Mandapati"
    }, 
    {
      "affiliation": "Complex and Distributed IT-Systems Group, TU Berlin, Berlin, Germany", 
      "@type": "Person", 
      "name": "Soeren Becker"
    }, 
    {
      "affiliation": "Huawei Munich Research Center, Munich, Germany", 
      "@type": "Person", 
      "name": "Jorge Cardoso"
    }, 
    {
      "affiliation": "Complex and Distributed IT-Systems Group, TU Berlin, Berlin, Germany", 
      "@type": "Person", 
      "name": "Odej Kao"
    }
  ], 
  "url": "https://zenodo.org/record/3549604", 
  "datePublished": "2019-10-14", 
  "version": "1.1", 
  "keywords": [
    "aiops", 
    "observability", 
    "dataset", 
    "tracing", 
    "metrics", 
    "logs", 
    "anomaly detection", 
    "root-cause analysis", 
    "distributed systems"
  ], 
  "@context": "https://schema.org/", 
  "distribution": [
    {
      "contentUrl": "https://zenodo.org/api/files/c96abc93-56d5-4e6d-a24b-0f2e0c9cafb3/concurrent data.zip", 
      "encodingFormat": "zip", 
      "@type": "DataDownload"
    }, 
    {
      "contentUrl": "https://zenodo.org/api/files/c96abc93-56d5-4e6d-a24b-0f2e0c9cafb3/sequential_data.zip", 
      "encodingFormat": "zip", 
      "@type": "DataDownload"
    }
  ], 
  "identifier": "https://doi.org/10.5281/zenodo.3549604", 
  "@id": "https://doi.org/10.5281/zenodo.3549604", 
  "@type": "Dataset", 
  "name": "Multi-Source Distributed System Data for AI-powered Analytics"
}
3,093
622
views
downloads
All versions This version
Views 3,0932,585
Downloads 622581
Data volume 249.8 GB189.5 GB
Unique views 2,1391,827
Unique downloads 241217

Share

Cite as