Network traffic datasets with novel extended IP flow called NetTiSA flow

Network traffic datasets with novel extended IP flow called NetTiSA flow

Datasets were created for the paper: NetTiSA: Extended IP Flow with Time-series Features for Universal Bandwidth-constrained High-speed Network Traffic Classification -- Josef Koumar, Karel Hynek, Jaroslav Pešek, Tomáš Čejka -- which is published in The International Journal of Computer and Telecommunications Networking https://doi.org/10.1016/j.comnet.2023.110147

Please cite the usage of our datasets as:

Josef Koumar, Karel Hynek, Jaroslav Pešek, Tomáš Čejka, "NetTiSA: Extended IP flow with time-series features for universal bandwidth-constrained high-speed network traffic classification", Computer Networks, Volume 240, 2024, 110147, ISSN 1389-1286

@article{KOUMAR2024110147,
title = {NetTiSA: Extended IP flow with time-series features for universal bandwidth-constrained high-speed network traffic classification},
journal = {Computer Networks},
volume = {240},
pages = {110147},
year = {2024},
issn = {1389-1286},
doi = {https://doi.org/10.1016/j.comnet.2023.110147},
url = {https://www.sciencedirect.com/science/article/pii/S1389128623005923},
author = {Josef Koumar and Karel Hynek and Jaroslav Pešek and Tomáš Čejka}
}

This Zenodo repository contains 23 datasets created from 15 well-known published datasets, which are cited in the table below. Each dataset contains the NetTiSA flow feature vector.

 

NetTiSA flow feature vector

The novel extended IP flow called NetTiSA (Network Time Series Analysed) flow contains a universal bandwidth-constrained feature vector consisting of 20 features. We divide the NetTiSA flow classification features into three groups by computation. The first group of features is based on classical bidirectional flow information---a number of transferred bytes, and packets.  The second group contains statistical and time-based features calculated using the time-series analysis of the packet sequences. The third type of features can be computed from the previous groups (i.e., on the flow collector) and improve the classification performance without any impact on the telemetry bandwidth.

Flow features

The flow features are:

• Packets is the number of packets in the direction from the source to the destination IP address.
• Packets in reverse order is the number of packets in the direction from the destination to the source IP address.
• Bytes is the size of the payload in bytes transferred in the direction from the source to the destination IP address.
• Bytes in reverse order is the size of the payload in bytes transferred in the direction from the destination to the source IP address.

Statistical and Time-based features

The features that are exported in the extended part of the flow. All of them can be computed (exactly or in approximative) by stream-wise computation, which is necessary for keeping memory requirements low. The second type of feature set contains the following features:

• Mean represents mean of the payload lengths of packets
• Min is the minimal value from payload lengths of all packets in a flow
• Max is the maximum value from payload lengths of all packets in a flow
• Standard deviation is a measure of the variation of payload lengths from the mean payload length
• Root mean square is the measure of the magnitude of payload lengths of packets
• Average dispersion is the average absolute difference between each payload length of the packet and the mean value
• Kurtosis is the measure describing the extent to which the tails of a distribution differ from the tails of a normal distribution
• Mean of relative times is the mean of the relative times which is a sequence defined as \(st = \{t_1 - t_1, t_2 - t_1, ..., t_n - t_1\} \)
• Mean of time differences is the mean of the time differences which is a sequence defined as \(dt = \{ t_j - t_i | j = i + 1, i \in \{1, 2, \dots, n - 1\} \}.\)
• Min from time differences is the minimal value from all time differences, i.e., min space between packets.
• Max from time differences is the maximum value from all time differences, i.e., max space between packets.
• Time distribution describes the deviation of time differences between individual packets within the time series. The feature is computed by the following equation:
  \(tdist = \frac{ \frac{1}{n-1} \sum_{i=1}^{n-1} \left| \mu_{\{dt_{n-1}\}} - dt_i \right| }{ \frac{1}{2} \left(max\left(\{dt_{n-1}\}\right) - min\left(\{dt_{n-1}\}\right) \right) }\)
• Switching ratio represents a value change ratio (switching) between payload lengths. The switching ratio is computed by equation:
  \(sr = \frac{s_n}{\frac{1}{2} (n - 1)}\)

           where \(s_n\) is number of switches.

Features computed at the collector
The third set contains features that are computed from the previous two groups prior to classification. Therefore, they do not influence the network telemetry size and their computation does not put additional load to resource-constrained flow monitoring probes. The NetTiSA flow combined with this feature set is called the Enhanced NetTiSA flow and contains the following features:

• Max minus min  is the difference between minimum and maximum payload lengths
• Percent deviation is the dispersion of the average absolute difference to the mean value
• Variance is the spread measure of the data from its mean
• Burstiness is the degree of peakedness in the central part of the distribution
• Coefficient of variation is a dimensionless quantity that compares the dispersion of a time series to its mean value and is often used to compare the variability of different time series that have different units of measurement
• Directions describe a percentage ratio of packet direction computed as \(\frac{d_1}{ d_1 + d_0}\), where \(d_1\) is a number of packets in a direction from source to destination IP address and \(d_0\) the opposite direction. Both  \(d_1\) and \(d_0\) are inside the classical bidirectional flow.
• Duration is the duration of the flow

The NetTiSA flow is implemented into IP flow exporter ipfixprobe.

Description of dataset files

In the following table is a description of each dataset file: