Dataset Open Access
AIT Log Data Set V2.0
Landauer, Max; Skopik, Florian; Frank, Maximilian; Hotwagner, Wolfgang; Wurzenberger, Markus; Rauber, Andreas
MARC21 XML Export
<?xml version='1.0' encoding='UTF-8'?>
<record xmlns="http://www.loc.gov/MARC21/slim">
<leader>00000nmm##2200000uu#4500</leader>
<datafield tag="653" ind1=" " ind2=" ">
<subfield code="a">log data</subfield>
</datafield>
<datafield tag="653" ind1=" " ind2=" ">
<subfield code="a">intrusion detection</subfield>
</datafield>
<controlfield tag="005">20220928133455.0</controlfield>
<datafield tag="500" ind1=" " ind2=" ">
<subfield code="a">M. Landauer, F. Skopik, M. Frank, W. Hotwagner, M. Wurzenberger, and A. Rauber. "Maintainable Log Datasets for Evaluation of Intrusion Detection Systems". arXiv:2203.08580</subfield>
</datafield>
<controlfield tag="001">5789064</controlfield>
<datafield tag="700" ind1=" " ind2=" ">
<subfield code="u">AIT Austrian Institute of Technology</subfield>
<subfield code="a">Skopik, Florian</subfield>
</datafield>
<datafield tag="700" ind1=" " ind2=" ">
<subfield code="u">AIT Austrian Institute of Technology</subfield>
<subfield code="a">Frank, Maximilian</subfield>
</datafield>
<datafield tag="700" ind1=" " ind2=" ">
<subfield code="u">AIT Austrian Institute of Technology</subfield>
<subfield code="a">Hotwagner, Wolfgang</subfield>
</datafield>
<datafield tag="700" ind1=" " ind2=" ">
<subfield code="u">AIT Austrian Institute of Technology</subfield>
<subfield code="a">Wurzenberger, Markus</subfield>
</datafield>
<datafield tag="700" ind1=" " ind2=" ">
<subfield code="u">Vienna University of Technology</subfield>
<subfield code="a">Rauber, Andreas</subfield>
</datafield>
<datafield tag="856" ind1="4" ind2=" ">
<subfield code="s">15837626707</subfield>
<subfield code="z">md5:b6cd133398fbed6f3e7434dafbc45756</subfield>
<subfield code="u">https://zenodo.org/record/5789064/files/fox.zip</subfield>
</datafield>
<datafield tag="856" ind1="4" ind2=" ">
<subfield code="s">16773705987</subfield>
<subfield code="z">md5:4fbe3dd17a194776b2851116f8b51380</subfield>
<subfield code="u">https://zenodo.org/record/5789064/files/harrison.zip</subfield>
</datafield>
<datafield tag="856" ind1="4" ind2=" ">
<subfield code="s">7132670599</subfield>
<subfield code="z">md5:78e9b7d169b438f03816019b52ab3ff7</subfield>
<subfield code="u">https://zenodo.org/record/5789064/files/russellmitchell.zip</subfield>
</datafield>
<datafield tag="856" ind1="4" ind2=" ">
<subfield code="s">10004843924</subfield>
<subfield code="z">md5:fbd2663a41c83f345aa08a7fb11a6c6c</subfield>
<subfield code="u">https://zenodo.org/record/5789064/files/santos.zip</subfield>
</datafield>
<datafield tag="856" ind1="4" ind2=" ">
<subfield code="s">17594549155</subfield>
<subfield code="z">md5:872bbec7ffe12f14e06364e328a96088</subfield>
<subfield code="u">https://zenodo.org/record/5789064/files/shaw.zip</subfield>
</datafield>
<datafield tag="856" ind1="4" ind2=" ">
<subfield code="s">17163231368</subfield>
<subfield code="z">md5:ff524634e64c7779def2eb804e8c52ac</subfield>
<subfield code="u">https://zenodo.org/record/5789064/files/wardbeck.zip</subfield>
</datafield>
<datafield tag="856" ind1="4" ind2=" ">
<subfield code="s">19593129576</subfield>
<subfield code="z">md5:e3f5605d867fcc644f93ae6c736cdb30</subfield>
<subfield code="u">https://zenodo.org/record/5789064/files/wheeler.zip</subfield>
</datafield>
<datafield tag="856" ind1="4" ind2=" ">
<subfield code="s">26544770737</subfield>
<subfield code="z">md5:952dd37cb262ef0054ad6d5371bdbe70</subfield>
<subfield code="u">https://zenodo.org/record/5789064/files/wilson.zip</subfield>
</datafield>
<datafield tag="542" ind1=" " ind2=" ">
<subfield code="l">open</subfield>
</datafield>
<datafield tag="260" ind1=" " ind2=" ">
<subfield code="c">2022-02-24</subfield>
</datafield>
<datafield tag="909" ind1="C" ind2="O">
<subfield code="p">openaire_data</subfield>
<subfield code="o">oai:zenodo.org:5789064</subfield>
</datafield>
<datafield tag="100" ind1=" " ind2=" ">
<subfield code="u">AIT Austrian Institute of Technology</subfield>
<subfield code="a">Landauer, Max</subfield>
</datafield>
<datafield tag="245" ind1=" " ind2=" ">
<subfield code="a">AIT Log Data Set V2.0</subfield>
</datafield>
<datafield tag="536" ind1=" " ind2=" ">
<subfield code="c">833456</subfield>
<subfield code="a">A cybersecurity framework to GUArantee Reliability and trust for Digital service chains</subfield>
</datafield>
<datafield tag="540" ind1=" " ind2=" ">
<subfield code="u">https://creativecommons.org/licenses/by-nc-sa/4.0/legalcode</subfield>
<subfield code="a">Creative Commons Attribution Non Commercial Share Alike 4.0 International</subfield>
</datafield>
<datafield tag="650" ind1="1" ind2="7">
<subfield code="a">cc-by</subfield>
<subfield code="2">opendefinition.org</subfield>
</datafield>
<datafield tag="520" ind1=" " ind2=" ">
<subfield code="a"><p><strong>AIT Log Data Sets</strong></p>
<p>This repository contains synthetic log data suitable for evaluation of intrusion detection systems, federated learning, and alert aggregation. A detailed description of the dataset is available in [1]. The logs were collected from eight testbeds that were built at the Austrian Institute of Technology (AIT) following the approach by [2]. Please cite these papers if the data is used for academic publications.</p>
<p>In brief, each of the datasets corresponds to a testbed representing a small enterprise network including mail server, file share, WordPress server, VPN, firewall, etc. Normal user behavior is simulated to generate background noise over a time span of 4-6 days. At some point, a sequence of attack steps are launched against the network. Log data is collected from all hosts and includes Apache access and error logs, authentication logs, DNS logs, VPN logs, audit logs, Suricata logs, network traffic packet captures, horde logs, exim logs, syslog, and system monitoring logs. Separate ground truth files are used to label events that are related to the attacks. Compared to the <a href="https://zenodo.org/record/4264796">AIT-LDSv1.1</a>, a more complex network and diverse user behavior is simulated, and logs are collected from all hosts in the network. If you are only interested in network traffic analysis, we also provide the <a href="https://zenodo.org/record/6610489">AIT-NDS</a> containing the labeled netflows of the testbed networks.</p>
<p>The datasets in this repository have the following structure:</p>
<ul>
<li>The <em>gather </em>directory contains all logs collected from the testbed. Logs collected from each host are located in <em>gather/&lt;host_name&gt;/logs/</em>.</li>
<li>The <em>labels </em>directory contains the ground truth of the dataset that indicates which events are related to attacks. The directory mirrors the structure of the gather directory so that each label files is located at the same path and has the same name as the corresponding log file. Each line in the label files references the log event corresponding to an attack by the line number counted from the beginning of the file (&quot;line&quot;), the labels assigned to the line that state the respective attack step (&quot;labels&quot;), and the labeling rules that assigned the labels (&quot;rules&quot;).</li>
<li>The <em>processing </em>directory contains the source code that was used to generate the labels.</li>
<li>The <em>rules </em>directory contains the labeling rules.</li>
<li>The <em>environment </em>directory contains the source code that was used to deploy the testbed and run the simulation using the <a href="https://github.com/ait-aecid/kyoushi-environment">Kyoushi Testbed Environment</a>.</li>
<li>The <em>dataset.yml</em> file specifies the start and end time of the simulation.</li>
</ul>
<p>The following table summarizes relevant properties of the datasets:</p>
<table align="center">
<thead>
<tr>
<th scope="row">Dataset</th>
<th scope="col">Simulation time</th>
<th scope="col">Attack time</th>
<th scope="col">Exfiltration visible in DNS logs</th>
<th scope="col">Scan volume</th>
<th scope="col">Password cracking</th>
<th scope="col">Unpacked size</th>
</tr>
</thead>
<tbody>
<tr>
<th scope="row">fox</th>
<td>2022-01-15 00:00 - 2022-01-20 00:00</td>
<td>2022-01-18 11:59 - 2022-01-18 13:15</td>
<td>Yes</td>
<td>High</td>
<td>Yes</td>
<td>26 GB</td>
</tr>
<tr>
<th scope="row">harrison</th>
<td>2022-02-04 00:00 - 2022-02-09 00:00</td>
<td>2022-02-08 07:07 - 2022-02-08 08:38</td>
<td>Yes</td>
<td>High</td>
<td>Yes</td>
<td>27 GB</td>
</tr>
<tr>
<th scope="row">russellmitchell</th>
<td>2022-01-21 00:00 - 2022-01-25 00:00</td>
<td>2022-01-24 03:01 - 2022-01-24 04:39</td>
<td>Yes</td>
<td>Low</td>
<td>Yes</td>
<td>14 GB</td>
</tr>
<tr>
<th scope="row">santos</th>
<td>2022-01-14 00:00 - 2022-01-18 00:00</td>
<td>2022-01-17 11:15 - 2022-01-17 11:59</td>
<td>Yes</td>
<td>Low</td>
<td>Yes</td>
<td>17 GB</td>
</tr>
<tr>
<th scope="row">shaw</th>
<td>2022-01-25 00:00 - 2022-01-31 00:00</td>
<td>2022-01-29 14:37 - 2022-01-29 15:21</td>
<td>No</td>
<td>Low</td>
<td>Yes</td>
<td>27 GB</td>
</tr>
<tr>
<th scope="row">wardbeck</th>
<td>2022-01-19 00:00 - 2022-01-24 00:00</td>
<td>2022-01-23 12:10 - 2022-01-23 12:56</td>
<td>Yes</td>
<td>Low</td>
<td>Yes</td>
<td>26 GB</td>
</tr>
<tr>
<th scope="row">wheeler</th>
<td>2022-01-26 00:00 - 2022-01-31 00:00</td>
<td>2022-01-30 07:35 - 2022-01-30 17:53</td>
<td>Yes</td>
<td>High</td>
<td>No</td>
<td>30 GB</td>
</tr>
<tr>
<th scope="row">wilson</th>
<td>2022-02-03 00:00 - 2022-02-09 00:00</td>
<td>2022-02-07 10:57 - 2022-02-07 11:49</td>
<td>Yes</td>
<td>High</td>
<td>Yes</td>
<td>39 GB</td>
</tr>
</tbody>
</table>
<p>The following attacks are launched in the network:</p>
<ul>
<li>Scans (nmap, WPScan, dirb)</li>
<li>Webshell upload (CVE-2020-24186)</li>
<li>Password cracking (John the Ripper)</li>
<li>Privilege escalation</li>
<li>Remote command execution</li>
<li>Data exfiltration (DNSteal)</li>
</ul>
<p>Note that attack parameters and their execution orders vary in each dataset. Labeled log files are trimmed to the simulation time to ensure that their labels (which reference the related event by the line number in the file) are not misleading. Other log files, however, also contain log events generated before or after the simulation time and may therefore be affected by testbed setup or data collection. It is therefore recommended to only consider logs with timestamps within the simulation time for analysis.</p>
<p>Beside the attack labels, a general overview of the exact times when specific attack steps are launched are available in <em>gather/attacker_0/logs/attacks.log</em>. An enumeration of all hosts and their IP addresses is stated in processing/config/servers.yml. Moreover, configurations of each host are provided in <em>gather/&lt;host_name&gt;/configs/</em> and <em>gather/&lt;host_name&gt;/facts.json</em>.</p>
<p>Version history:</p>
<ul>
<li><a href="https://doi.org/10.5281/zenodo.3723082">AIT-LDS-v1.x</a>: Four datasets, logs from single host, fine-granular audit logs, mail/CMS.</li>
<li><a href="http://doi.org/10.5281/zenodo.5789064">AIT-LDS-v2.0</a>: Eight datasets, logs from all hosts, system logs and network traffic, mail/CMS/cloud/web.</li>
</ul>
<p>Acknowledgements: Partially funded by the FFG projects INDICAETING (868306) and DECEPT (873980), and the EU projects GUARD (833456) and PANDORA (SI2.835928).</p>
<p><strong>If you use the dataset, please cite the following publications:</strong></p>
<p>[1] M. Landauer, F. Skopik, M. Frank, W. Hotwagner, M. Wurzenberger, and A. Rauber. &quot;Maintainable Log Datasets for Evaluation of Intrusion Detection Systems&quot;. Under Review. <a href="https://arxiv.org/abs/2203.08580">arXiv:2203.08580</a> [<a href="https://arxiv.org/pdf/2203.08580.pdf">PDF</a>]</p>
<p>[2]&nbsp;M. Landauer, F. Skopik, M. Wurzenberger, W. Hotwagner and A. Rauber, <a href="https://ieeexplore.ieee.org/document/9262078">&quot;Have it Your Way: Generating Customized Log Datasets With a Model-Driven Simulation Testbed,&quot;</a> in IEEE Transactions on Reliability, vol. 70, no. 1, pp. 402-415, March 2021, doi: 10.1109/TR.2020.3031317. [<a href="https://www.skopik.at/ait/2020_trel.pdf">PDF</a>]</p></subfield>
</datafield>
<datafield tag="773" ind1=" " ind2=" ">
<subfield code="n">doi</subfield>
<subfield code="i">isVersionOf</subfield>
<subfield code="a">10.5281/zenodo.5789063</subfield>
</datafield>
<datafield tag="024" ind1=" " ind2=" ">
<subfield code="a">10.5281/zenodo.5789064</subfield>
<subfield code="2">doi</subfield>
</datafield>
<datafield tag="980" ind1=" " ind2=" ">
<subfield code="a">dataset</subfield>
</datafield>
</record>
1,530
9,579
views
downloads
| All versions | This version | |
|---|---|---|
| Views | 1,530 | 1,530 |
| Downloads | 9,579 | 9,579 |
| Data volume | 151.5 TB | 151.5 TB |
| Unique views | 1,311 | 1,311 |
| Unique downloads | 651 | 651 |
Indexed in
- Publication date:
- February 24, 2022
- DOI:
-
- Keyword(s):
- log data intrusion detection
- Grants:
-
European Commission:
- GUARD - A cybersecurity framework to GUArantee Reliability and trust for Digital service chains (833456)
- License (for files):
- Creative Commons Attribution Non Commercial Share Alike 4.0 International