CESNET-TLS-Year22: A year-spanning TLS network traffic dataset from backbone lines

We recommend using the CESNET DataZoo python library, which facilitates the work with large network traffic datasets. More information about the DataZoo project can be found in the GitHub repository https://github.com/CESNET/cesnet-datazoo.

The modern approach for network traffic classification (TC), which is an important part of operating and securing networks, is to use machine learning (ML) models that are able to learn intricate relationships between traffic characteristics and communicating applications. A crucial prerequisite is having representative datasets. However, datasets collected from real production networks are not being published in sufficient numbers. Thus, this paper presents a novel dataset, CESNET-TLS-Year22, that captures the evolution of TLS traffic in an ISP network over a year. The dataset contains 180 web service labels and standard TC features, such as packet sequences. The unique year-long time span enables comprehensive evaluation of TC models and assessment of their robustness in the face of the ever-changing environment of production networks.

Data description The dataset consists of network flows describing encrypted TLS communications. Flows are extended with packet sequences, histograms, and fields extracted from the TLS ClientHello message, which is transmitted in the first packet of the TLS connection handshake. The most important extracted handshake field is the SNI domain, which is used for ground-truth labeling. 

Packet Sequences Sequences of packet sizes, directions, and inter-packet times are standard data input for traffic analysis. For packet sizes, we consider the payload size after transport headers (TCP headers for the TLS case). We omit packets with no TCP payload, for example ACKs, because zero-payload packets are related to the transport layer internals rather than services’ behavior. Packet directions are encoded as ±1, where +1 means a packet sent from client to server, and -1 is a packet from server to client. Inter-packet times depend on the location of communicating hosts, their distance, and on the network conditions on the path. However, it is still possible to extract relevant information that correlates with user interactions and, for example, with the time required for an API/server/database to process the received data and generate a response. Packet sequences have a maximum length of 30, which is the default setting of the used flow exporter. We also derive three fields from each packet sequence: its length, time duration, and the number of roundtrips. The roundtrips are counted as the number of changes in the communication direction; in other words, each client request and server response pair counts as one roundtrip.

Flow statistics Each data record also includes standard flow statistics, representing aggregated information about the entire bidirectional connection. The fields are the number of transmitted bytes and packets in both directions, the duration of the flow, and packet histograms. The packet histograms include binned counts (not limited to the first 30 packets) of packet sizes and inter-packet times in both directions. There are eight bins with a logarithmic scale; the intervals are 0-15, 16-31, 32-63, 64-127, 128-255, 256-511, 512-1024, >1024 [ms or B]. The units are milliseconds for inter-packet times and bytes for packet sizes (More information in the PHISTS plugin documentation). Moreover, each flow has its end reason---either it ended with the TCP connection termination (FIN packets), was idle, reached the active timeout, or ended due to other reasons. This corresponds with the official IANA IPFIX-specified values. The FLOW_ENDREASON_OTHER field represents the forced end and lack of resources reasons.

Dataset structure The dataset is organized per weeks and individual days. The flows are delivered in compressed CSV files. CSV files contain one flow per row; data columns are summarized in the provided list below. For each flow data file, there is a JSON file with the total number of saved flows and the number of flows per service. There are also files aggregating flow counts for each week (stats-week.json) and for the entire dataset (stats-dataset.json).  The following list describes flow data fields in CSV files: 

• ID: Unique identifier
• SRC_IP: Source IP address
• DST_IP: Destination IP address
• DST_ASN: Destination Autonomous System number
• SRC_PORT: Source port
• DST_PORT: Destination port
• PROTOCOL: Transport protocol
• FLAG_CWR: Presence of the CWR flag
• FLAG_CWR_REV: Presence of the CWR flag in the reverse direction
• FLAG_ECE: Presence of the ECE flag
• FLAG_ECE_REV: Presence of the ECE flag in the reverse direction
• FLAG_URG: Presence of the URG flag
• FLAG_URG_REV: Presence of the URG flag in the reverse direction
• FLAG_ACK: Presence of the ACK flag
• FLAG_ACK_REV: Presence of the ACK flag in the reverse direction
• FLAG_PSH: Presence of the PSH flag
• FLAG_PSH_REV: Presence of the PSH flag in the reverse direction
• FLAG_RST: Presence of the RST flag
• FLAG_RST_REV: Presence of the RST flag in the reverse direction
• FLAG_SYN: Presence of the SYN flag
• FLAG_SYN_REV: Presence of the SYN flag in the reverse direction
• FLAG_FIN: Presence of the FIN flag
• FLAG_FIN_REV: Presence of the FIN flag in the reverse direction
• TLS_SNI: Server Name Indication domain
• TLS_JA3: JA3 fingerprint of TLS client
• TIME_FIRST: Timestamp of the first packet in format YYYY-MM-DDTHH-MM-SS.ffffff
• TIME_LAST: Timestamp of the last packet in format YYYY-MM-DDTHH-MM-SS.ffffff
• DURATION: Duration of the flow in seconds
• BYTES: Number of transmitted bytes from client to server
• BYTES_REV: Number of transmitted bytes from server to client
• PACKETS: Number of packets transmitted from client to server
• PACKETS_REV: Number of packets transmitted from server to client
• PPI: Packet sequence in the format: [[inter-packet times], [packet directions], [packet sizes]]
• PPI_LEN: Number of packets in the PPI sequence
• PPI_DURATION: Duration of the PPI sequence in seconds
• PPI_ROUNDTRIPS: Number of roundtrips in the PPI sequence
• PHIST_SRC_SIZES: Histogram of packet sizes from client to server
• PHIST_DST_SIZES: Histogram of packet sizes from server to client
• PHIST_SRC_IPT: Histogram of inter-packet times from client to server
• PHIST_DST_IPT: Histogram of inter-packet times from server to client
• APP: Web service label
• CATEGORY: Service category
• FLOW_ENDREASON_IDLE: Flow was terminated because it was idle
• FLOW_ENDREASON_ACTIVE: Flow was terminated because it reached the active timeout
• FLOW_ENDREASON_END: Flow ended with the TCP connection termination
• FLOW_ENDREASON_OTHER: Flow was terminated for other reasons