Evaluation of Apache Spot's machine learning capabilities in an SDN/NFV enabled environment

Software Defined Networking (SDN) and Network Function Virtualisation (NFV) are transforming modern networks towards a service-oriented architecture. At the same time, the cybersecurity industry is rapidly adopting Machine Learning (ML) algorithms to improve detection and mitigation of complex attacks. Traditional intrusion detection systems perform signature-based detection, based on well-known malicious traffic patterns that signify potential attacks. The main drawback of this method is that attack patterns need to be known in advance and signatures must be preconfigured. Hence, typical systems fail to detect a zero-day attack or an attack with unknown signature. This work considers the use of machine learning for advanced anomaly detection, and specifically deploys the Apache Spot ML framework on an SDN/NFV-enabled testbed running cybersecurity services as Virtual Network Functions (VNFs). VNFs are used to capture traffic for ingestion by the ML algorithm and apply mitigation measures in case of a detected anomaly. Apache Spot utilises Latent Dirichlet Allocation to identify anomalous traffic patterns in Netflow, DNS and proxy data. The overall performance of Apache Spot is evaluated by deploying Denial of Service (Slowloris, BoNeSi) and a Data Exfiltration attack (iodine).


INTRODUCTION
The increasing demand for high-speed internet access has led to the evolution of the Internet towards softwarised and more scalable architectures.Software Defined Networking (SDN) [1] and Network Function Virtualisation (NFV) [2] are considered two key enabling technologies that underline the evolution of future infrastructures.The SDN/NFV paradigm enables ISPs to compose and manage complex network services at the click of a button, while drastically improving time and cost efficiency [3].
At the same time, the growing number of complex cyber threats is often overlooked by traditional cybersecurity systems, due to the size, variety and velocity of traffic data that need to be inspected for signs of malicious activity.Cyber-attacks are observed with increasing frequency and impact radius, even on global scale [4].With that in mind, governments and organizations are increasing their cybersecurity investment to ensure the integrity of their data and networks, focusing on advanced systems for detection and mitigation.Signature-based intrusion detection systems [5] search for known suspicious patterns that are often attributed to known cyber-attacks.The obvious drawback is that the attack signatures need to be known in advance and the detection techniques are too rigid, making detection harder in case of a multi-vector attack or zero-day exploit.Anomaly detection systems [6], on the other hand, can be trained with typical network traffic to create a baseline of normal traffic.This is especially critical when there is no attack signature associated with the malicious activity.Misclassified results, such as false positive or false negative detection, however, can have adverse operational effects.
Previous research in this area has considered a variety of different ML algorithms for anomaly detection.The authors in [7] follow a more general approach and present research related to anomaly detection, without focusing on its practical use in network security.The work in [8] and [9] focuses on surveying intrusion detection in the context of cyber security through data mining and machine learning methods.A multitude of methods have been used for the implementation of machine learning classifiers in cyber security research for anomaly detection.One approach is that of Artificial Neural Networks (ANN) [10] that requires a more time-consuming preprocessing stage, an ANN training stage and an ANN decision stage.Another popular approach is the use of a Bayesian network [11]; in that work, Bayesian networks are used to classify operating system calls which are the result of the reception of TCP/IP packets.The EXPOSURE system [12] utilizes decision trees in order to perform passive DNS analysis.Other classifiers, like Random Forest [13] are combinations of decision trees and ensemble learning, while Naïve Bayes [14], approaches offer a simplistic implementation of a Bayes classifier.
This work considers the case of Apache Spot [15], a machine learning (ML)-based platform for anomaly detection, that utilises Latent Dirichlet Allocation [16] to detect unusual traffic patterns.Latent Dirichlet Allocation is a Natural Language Processing algorithm, which is a factor that sets it apart from the common machine learning classifiers in use.Natural Language Processing methods are easy to apply on the variety of different network traffic logs and improve the overall threat intelligence capabilities by including more sources of structured, humanreadable, textual data.
Other systems offer similar capabilities, such as Sqrrl [17] and Apache Metron [18]; Apache Spot sets itself apart not only by its machine learning capabilities, but also from its Open Data Model (ODM).As machine learning becomes a mainstream technology that is present in many consumer products, the cybersecurity industry has been quick to adopt it to improve on the existing defense capabilities.ODM brings together all security-related data (event, user, network, endpoint, etc.) into a singular view that can be used to detect threats more effectively.It also provides the ability to share and reuse threat detection models, analytics, and more.This improves interoperability among anomaly detection platforms and fosters the creation of an open data community.
Apache Spot is deployed and evaluated in the SHIELD [19] [20] SDN/NFV-enabled testbed in Athens, Greece.SHIELD proposes "a universal solution for dynamically establishing and deploying virtual security infrastructures into ISP and corporate networks".The project builds on the huge momentum of Network Functions Virtualisation (NFV) in order to virtualise security appliances into virtual Network Security Functions (vNSFs), to be instantiated within the network infrastructure.
Spot's integration with SDN/NFV leads to improved data ingestion, while cybersecurity VNFs can immediately receive threat information and apply mitigation measures.Its overall performance is tested using various penetration testing tools.In this work UDP flooding [21], Slowloris [21] and DNS Tunneling [22] attacks are performed.These attacks were selected due to the variation in the attack traffic in terms of velocity, variety and volume.In the case of UDP flooding, the attack traffic features high volume and velocity.Slowloris is a protocol-based Denial of Service attack that generates much smaller amounts of malicious incoming traffic.DNS tunneling is used as an example of a DNSbased data exfiltration attack that generates small amounts of traffic.
The rest of this paper is organized as follows: Section 2 presents the SDN/NFV testbed and Spot's core functions, deployment and mode of operation.It also presents the penetration testing tools used to perform the attacks against the network.Section 3 analyses the results of the penetration tests.Section 4 concludes this paper and discusses future work.The presented analysis aims to explore the performance of Apache Spot in this network setup, evaluate machine learning-based attack detection, and investigate Apache Spot configuration optimization for improvement of its performance in an SDN/NFV context.

The SHIELD Environment
SHIELD aims to develop a next-generation cybersecurity platform tailored for software networks, based on the SDN/NFV paradigm, big data and infrastructure attestation.The core components of the SHIELD platform (Figure 1) are deployed in two SDN/NFV testbeds in Athens and Barcelona, and include [23]  The DARE is an information-driven intrusion detection and prevention platform that stores and analyses heterogeneous network information, previously collected via the vNSFs.

 The virtual Network Security Functions (vNSFs):
OSM deploys cybersecurity as-a-Service in the form of KVM-based vNSFs (such as firewalls, deep packet inspection, intrusion detection etc.)


The vNSF Store: Registers valid vNSF images from vNSF developers.


The Trust Monitor [28]: Integrity of the NFVI, the vNSFs and the SDN Controller is checked periodically to detect compromised software and/or hardware.This work is based on the Trusted Computing paradigm and its Remote Attestation workflow.


The Security dashboard and controller: The dashboard provides an overview of the security status and allows the operators to apply remediation actions received by the remediation engine of the DARE.The DARE features cognitive and analytical components capable of predicting specific vulnerabilities and attacks.The processing and analysis of large amounts of data is carried out using big data analytics and ML techniques.Furthermore, the DARE remediation engine utilises the outputs from the cognitive and data analytics modules and various contextual information to determine a mitigation plan for the existing threats.The mitigation actions are relayed to the vNSFO in the XML-based Medium-Level Security Policy Language (MSPL) format [29].The vNSFO then sends the appropriate rules to the vNSF, which applies them.For example, SHIELD deploys an OVS-based firewall vNSFs in a KVM/CentOS virtual machine, which receives rules to block IP addresses etc.This paper focuses on Apache Spot which is deployed as a DARE component

Apache Spot
Apache Spot is an open-source platform that aims to facilitate the detection of cyber threats by using machine learning, rather than traditional signature-based detection.Apache Spot collects NetFlow, DNS and proxy data and analyses them to learn what constitutes typical network traffic.Anomalous traffic patterns are then identified and reported.The following subsections present Spot's core functions (Figure 2).

Data Ingestion
The data ingestion framework follows a distributed architecture that minimises the possibility of data loss and ensures the availability of the service, even under heavy data loads that require significant processing power.
NetFlow, DNS and proxy data are captured in the network in nfcapd 1 , pcap 2

Machine Learning
Apache Spot's machine learning includes routines for analyzing suspicious connects from the collection of the previously ingested data, based on Latent Dirichlet Allocation (LDA) [16].LDA is a generative probabilistic model for collections of discrete data such as text corpora.Specifically, LDA is a topic modeling algorithm that defines a three-level hierarchical Bayesian model, in which each item of a collection is modeled as a finite mixture over an underlying set of topics.For its usage in Apache Spot it has been implemented using Spark MLlib [30].Apache Spot applies LDA to network traffic, by converting the log entries to words through aggregation and discretization.This way, documents correspond to IP addresses, words to log entries (corresponding to an IP address) and topics to normal network traffic profiles.As a result, Apache Spot deduces a probabilistic model for each IP's behavior.The model assigns to each log entry a probability.The events (log entries) with the lowest probabilities are flagged as suspicious for further analysis.The output of this analysis is a list which includes the events that are less probable to occur according to a model of normal traffic conditions.Less probability of encountering a specific type of traffic means that this traffic is considered by Spot as the most suspicious.

Operational Analytics & Semi-Supervised Learning
Apache Spot provides a graphical Operational Analytics (OA) interface where it analyzes and presents the results of the machine learning algorithm.The results are divided in NetFlow, DNS, Proxy and by date of creation.Apache Spot allows semisupervised learning through its OA module, in the sense that the user can evaluate and affect the model by scoring IP addresses and ports in terms of possible risks.These scores become available for use in the next execution of the ML algorithm.In this experimental deployment, no scoring was performed on the user-side.

Penetration Testing
The following attacks were utilized to assess Spot's performance: Slowloris, UDP Flooding and DNS Tunneling.

Slowloris
Slowloris [31] is a protocol-based DoS attack, that exploits HTTP protocol mechanisms to overwhelm the target, without the need to generate a massive flood of traffic.This attack establishes a lot of partial HTTP connections, that are not allowed to complete or to time-out.It is, therefore, a low bandwidth attack with almost no side-effects on other services and ports.By sending partial requests, Slowloris establishes connections with the target server which remain active as the server waits for the requests to complete, thus depleting its network resources.The server quickly reaches its maximum number of concurrent connections, which results in Denial of Service.The fact that this attack uses legitimate and not malformed packets, low bandwidth, and doesn't affect other services makes it very difficult for classic intrusion detection systems to detect.

UDP Flooding
BoNeSi [32] is a tool for simulating DDoS attacks.The attack is deployed using the default settings, which results to the creation of a UDP flood.The UDP protocol is connectionless and sessionless, which means that it establishes connections without the use of a three-way handshake like the TCP protocol.This makes it easy to abuse in order to overwhelm the target with a flood of UDP packets containing random data and it also means that it doesn't require a lot of resources to be committed by the attacker, other than network bandwidth, to be successful.The target server tries to find the service to which these packets are addressed to by reading the data and when it fails to find the service, it replies with "ICMP destination unreachable" packets.The UDP flood differs from Slowloris in the sense that it requires massive amounts of traffic to compromise the target server.

DNS Tunneling
The DNS tunneling [22] attack is used to bypass firewall restrictions and exfiltrate data from a network and/or establish a command and control channel between a commanding host outside the firewalled network perimeter and a compromised/accomplice host inside this perimeter.The data or the commands and their responses, are encoded in DNS queries and directed to a DNS server operated by the attacker.The malicious server is assigned a domain, so that any DNS query containing this domain is forwarded to the attacker.Iodine [33] is a tool for sending IPV4 data through a DNS server, by deploying a DNS tunnel among two endpoints.The iodine client serves as the compromised system that sends data through the tunnel, and the iodine server is deployed as the malicious authoritative DNS server that receives the exfiltrated data or handles the command and control channel.

Deployment Configuration
The installation of Apache Spot (Figure 3) was made on a Cloudera Cluster version 5.11.1y.A minimum of three Cloudera nodes is required in order to deploy the Setup, Ingestion, Machine Learning and Operational Analytics components of Apache Spot.Three hosts were set up running Ubuntu Trusty 14.04 and the prerequisite components (HDFS, Hive, Impala, Kafka, Spark, Yarn, and Zookeeper).Our setup with the 3 Virtual Machines (VMs) was the following:


The target server: The victim that is under Apache Spot's supervision.


The traffic generator: Replays files which contain normal, legitimate traffic.


The attacker: A Kali Linux VM, which performs the attacks against the victim webserver.
Network traffic was recorded and anonymised from the network of the Media Networks Laboratory (at the Institute of Informatics and Telecommunications of NCSR Demokritos) and saved in pcap files.For the NetFlow Telemetry, the target server forwards the traffic from its network interface, using the fprobe tool, to cloudera-host-2, which is the server responsible for the data ingestion function of Apache Spot.Cloudera-host-2 records the traffic in nfcapd files.The attacks are deployed through Kali Linux in real time (during the replay of the network traffic from the traffic generator).This way, they are also forwarded to cloudera-host-2 and saved in the aforementioned nfcapd files.
For the DNS telemetry, three virtual machines running Ubuntu 16.04 Server and one virtual machine running Ubuntu 14.04 Server were deployed and configured as follows:


The legitimate users 1 and 2: These users load wellknown webpages and save the traffic in pcap files, thus creating normal DNS traffic.


Iodine Client: Loads well-known websites, but additionally sends data using the iperf tool through a DNS tunnel which is established with the Iodine Server and these are saved in pcap files.


Iodine Server: Receives the data sent by the Iodine Client using the iperf tool at the network interface of the DNS tunnel.In the DNS tunneling scenario, PhantomJS [34] is used to create traffic towards Alexa's top 100 websites [35].This generates the normal DNS traffic for this experiment, which is saved in pcap files ingested by Apache Spot.The iodine client signifies the compromised system that has set up the tunnel.Iperf [36] traffic is sent through the tunnel, encoded in the DNS Each attack was executed ten times, and the results were collected and analyzed.The analysis results are presented and discussed in the following section.

RESULTS AND DISCUSSION
Apache Spot outputs the results of the ML algorithm in the form of a csv file.The csv file contains a ranking of the network traffic with the first entry being the one considered the most dangerous.Each entry contains the probability of being normal as calculated by the machine learning algorithm.Furthermore, each entry provides extensive information like date and time, source and destination IP addresses, source and destination ports, network protocol etc.The file is further processed to obtain the following metrics:


The attack ranking, meaning the position of the first entry which is part of the attack.This metric is labeled as "Position" in our results.


The probability assigned to the first entry which is part of the attack.This metric is labeled as "Probability" in our results.


The number of entries which are part of the attack in the first 100 entries of the CSV file.This metric is labeled as "First 100" in our results.


The percentage of the entries which are part of the attack for the whole CSV file.This metric is labeled as "Occurrence %" in our results.In order to get a better visualization of the distribution between the ten attacks executed with each penetration testing tool and determine the performance of the classifier, results are presented in boxplot and Receiver Operating Characteristic (ROC) curve.A receiver operating characteristic (ROC) curve [37] is a technique used for visualizing the performance of binary classifiers.The plot is created by using the True Positive Rate of the classifier, also known as sensitivity and the False Positive Rate, also known as Fall-Out.For example, if a given result x is a true positive the curve moves 1/(#-of-positives) upwards and if false positive 1/(#-of-negatives) to the right.The area under the ROC curve, called AUROC, represents the probability that the classifier will rank a randomly chosen true positive instance higher than a randomly chosen false positive instance.
The boxplot (also known as a box-and-whiskers plot) is a single dimensional figure that illustrates the overall properties of a selected dataset in a simple visualization.The lower boundary of the box represents the first quartile Q1, i.e. the boundary that denotes 25% of the measurements.The median (Q2) lies within the box, and the top boundary of the box represents the Q3 quartile (75% of the measurements).The relative position of the median, with respect to Q1 and Q3, can indicate whether a distribution is symmetrical or skewed.The interquartile range IQR is defined as: The system's range of normal operations is defined by the whiskers, which are calculated as a function of the interquartile range: Where k is a constant, set to k=1.5 as per Tukey's recommendation [38].If the lower and upper whisker exceed the range of the minimum and maximum value, they are set to match them.If there are measurements that lie outside of the range defined by the whiskers, they are considered outliers (Figure 4).

Slowloris
Table 1 illustrates experimental results after the ten executions of the Slowloris attack (in chronological order).The continuously open TCP connections make it stand out as an anomaly inside the data gathered from the network and thus it's being ranked as an anomaly.Although the first instance of the detected anomaly is ranked highly in terms of its position, the low number of anomalies in the first 100 results indicates the presence of numerous false positives.Slowloris is a low bandwidth attack taking up 2-4% of total traffic, as indicated by the low Occurrence percentage.A better indication of how efficiently Apache Spot detects the Slowloris attack is the high average AUROC value.AUROC evaluates the results of the machine learning model across the entire set of measurements.

BoNeSi
Table 2 depicts the results of ten executions of the BoNeSi attack (in chronological order).The BoNeSi attack was not successfully detected, which is attributed to the high occurrence percentage.The traffic generated by the UDP flood is much higher than normal traffic and thus disrupts the trained model.The algorithm starts considering the attack traffic as normal traffic rather than an anomaly.This is also supported by the low average AUROC value, which indicates that the BoNeSi traffic was considered as normal by the machine learning model.for the 75% of the distribution, but the remaining 25% reaches high values, thus the distribution is skewed.The BoNeSi box plot has a larger interquartile range showing that the results for the Position metric exhibit more variance in the case of BoNeSi.Figure 6 presents the box plots for the Probability metric for both Slowloris and BoNeSi.Similar values are observed up to the distribution's 75% percentile, although in the case for BoNeSi the top 25% measurements show significantly worse detection performance.The upper whisker in this case represents a useful metric and can be considered to be the threshold value for detection.Apache Spot allows the user to set a threshold value for anomaly detection and restrict the data that are analysed and visualized in its Operational Analytics.Considering that the overload of information that typically occurs in cyber security response, this is an important metric to estimate for a variety of cyber attacks.However, it is also shown that attacks with differing velocity and volume characteristics can vary in their detection thresholds.Figure 7 presents the ROC curve of the eighth Slowloris test as an example.Figure 8 illustrates the ROC curve of the second BoNeSi test as an example.

Iodine
Table 3 presents the results of the ten executions of the Iodine attack experiment, while Figure 9 illustrates the ROC curve of the second test as an example.In the case of iodine, the results vary, showing that Spot does not effectively detect this attack.The low AUROC value shows that the classifier would need additional intelligence to better distinguish between false and true positives.Metrics such as the length of the DNS packet or the information entropy for the tunneled traffic would be necessary to improve the detection capabilities of Apache Spot in the context of this attack.

Varying traffic conditions
In order to assess how traffic volume affects the detection capabilities of Apache Spot, additional tests were performed with Slowloris, with a three-fold increase in normal traffic (in terms of number of flows) and a shorter attack window that produced less malicious flows.In effect, the occurrence percentage of the attack was decreased, and more normal traffic was provided to the machine learning model.
In Table 4 and Figure 10, the ML algorithm ranked about 90% of the results with even lower probability, as it stood out more as an anomaly.The first 100 worst ranked results still contained a large number of false positives just as in the case of the initial setup.This increase can be explained by the lower occurrence of the attack, which makes the attack stand up as an anomaly.Another factor contributing to the increase of AUROC is the lower number of attack flows which results in an increase of the true positive rate (ROC).Figure 11 compares the probabilities assigned to normal legitimate traffic versus the Slowloris attack traffic.Normal traffic is shown to feature a wider range of operations (between the whiskers) which is expected, as it is ranked as more probable.The area in the boxplot near the normal traffic lower whisker, shows that normal and attack traffic probabilities overlap.This designates the area where the false positives may occur.Additional computation or an appropriate classification algorithm is necessary in this case, to reduce the number of false positive detections.Hence, while Apache Spot can effectively detect anomalies in the network traffic, it is not able to distinguish between atypical legitimate traffic and attack traffic without further effort.

CONCLUSIONS
In this work, the Apache Spot machine learning framework was deployed and assessed in terms of its performance regarding the detection of three different attacks, namely Slowloris, UDP flooding and DNS tunneling.Results show that Apache Spot can successfully detect attacks such as Slowloris, which generate small amounts of network traffic and usually evade common intrusion detection systems (IDS).The results were less promising in the case of UDP flooding and DNS tunneling.In the case of UDP flooding, the amount of attack traffic disrupted the trained model and caused many misclassifications.
Results showed that the velocity and volume of the attack traffic, with respect to the normal traffic on the network, are determining factors that affect its detection capabilities.The machine learning algorithm requires a large amount of normal traffic to be trained and create a reference model for normal traffic.As normal traffic increases, true positive rates increase as well.False positives occurred in all analyses more frequently than false negatives, although they were ranked with low probabilities, as top threats.The high false positive rate may indicate a vulnerability that can lead to the exploitation of MLbased systems to induce a Denial-of-Service by misclassifying legitimate traffic as anomalous and applying remediation measures against it.
The results were positively affected by the increase in normal traffic and the decrease of the attack traffic which lead to a lower occurrence of the attack.Based on these conclusions, our future work is steered towards estimating how the ML algorithms would be vulnerable to poisoning attacks that would cause the training model to be compromised and cause misclassifications, in the form of false positives and false negatives.

Germany 3 
[24]: Evaluation of Apache Spot's machine learning capabilities in an SDN/NFV enabled environment CyberTIM Workshop, ARES Conference 2018, August 2018, Hamburg, The vNSF Orchestrator (vNSFO): vNSFO is based on the ETSI-supported open-source Management and Orchestration framework (Open Source MANO).The OSM orchestrator [25] is used to manage the lifecycle of the vNSFs.OSM on-boards vNSF packages, instantiates Network Services (NSs) in specific points of presence within the network infrastructure and monitors the running services. The Virtual Infrastructure Manager (VIM): ETSI-MANO [26] defines the VIM as the framework that manages computing, storage and network resources.SHIELD deploys the OpenStack platform as a VIM. The SDN Controller: OpenDaylight Carbon [27] is used as the SDN controller. The Data Analysis and Remediation Engine (DARE): . The anomalies detected by Apache Spot are passed to the remediation engine, which provides specific rules to the SHIELD vNSFs, such as blocking or rate limiting rules etc.This paper focuses on the Athens experimentation testbed, which is deployed and maintained by the National Centre for Scientific Research "Demokritos" (NCSRD), Orion Innovations P.C. and Space Hellas S.A.The Athens testbed provides ETSI-compliant [2] Network Function Virtualisation Infrastructure (NFVI) and deploys vNSFs, such as virtualized Firewalls, Deep Packet Inspection, Intrusion Detection and an instance of Apache Spot.

Figure 3 :
Figure 3: Deployment configuration of the experimental setup.

Figure 5
Figure 5 presents the box plots for the Position metric for both Slowloris and BoNeSi.The Slowloris box plot shows good results

Figure 10 :
Figure 10: Probability metric for the Slowloris attack, initial vs. modified traffic flow pattern.

Figure 11 :
Figure 11: Probability metric for normal and attack traffic for the Slowloris attack, modified traffic flow pattern.
and bluecoat3formats, respectively.The corresponding files are aggregated in Spot's filesystem, which is monitored by Collector daemons.Spot's Collectors initiate the ingestion process when new data are found in the filesystem.At this point, the data are human-readable.After the collection of the data, Spot's Worker daemons ingest the data, convert them to Avro-Parquet format4and place them in a Hive database.Spark application to read data from Apache Kafka.The ingestion of the data in Hive tables at this point is necessary, since it prepares them in a form that is easily parsed by the Machine Learning algorithm.

Table 4 : Slowloris results for the modified traffic flow pattern.
Evaluation of Apache Spot's machine learning capabilities in an SDN/NFV enabled environment CyberTIM Workshop, ARES Conference 2018, August 2018, Hamburg, Germany