Wireless Interface Agent for SDN mmWave Multi-hop Networks: Design and Experimental Evaluation

Millimeter wave (mmwave) communications will likely be an enabler for 5G due to its multi-gigabit per second throughput capabilities. Furthermore,mmWave communications will have to be integrated in a new redesigned network required by 5G to fulfill its ambitious targets. In this paper, we present the design and implementation of a management agent for wireless devices deployed in a heterogeneous SDN wireless multi-hop research platform featuring mmwave communications for crosshauling (backhaul and fronthaul) purposes. The performance of the deployed mmwave network, based on the IEEE 802.11ad standard, is measured employing this agent. We measure the downtime in the presence of link up/down events, with obtained response times in the order of 10s-to-100s of milliseconds depending on the case. Furthermore, the TCP performance over the multi-hop 802.11ad mmwave network is also experimentally evaluated. In fact, TCP throughput up to around 800Mbps are obtained for single and multi-hop scenarios despite neighboring links using the same channel. Finally, one can also observe the impact of MTU size on TCP throughput, which may hinder the full exploitation of the mmWave link capacity when combined with other transport technologies, since the advantages of big MTUs (much bigger than the typical 1500 bytes) offered by mmwave devices may not be reaped.


INTRODUCTION
Upcoming 5G networks are addressing ambitious Key Performance Indicators (KPIs) in terms of not only overall/per user capacity and latency, but also, in terms of network control and management. Supporting these requirements allows 5G networks to expand their service scope to vertical industries, such as eHealth, automotive, media, or industry 4.0, hence enriching the telecom ecosystem.
On one hand, 5G rede nes the air interface and communications at millimeter wave (mmwave) bands (28-100 GHz frequency range) will likely to be an enabler to achieve the aforementioned objectives due to its multi-gigabit per second capacity thanks to the large chunks of contiguous available bandwidth. On the other hand, 5G networks will also require a redesign of the overall network architecture, where complexity and heterogeneity will be the rule, hence requiring increasing automation and programmability. Furthermore, the borders between traditional network segments (access, transport and core) will likely to be blurred and distributed computing resources will be deployed across the mobile network infrastructure. The consequent research problems of these network scenarios are tackled by EU H2020 5G-PPP phase 2 5GTRANSFORMER project [2]. In this sense, 5G embraces software de ned networking (SDN) and network function virtualization (NFV) principles as key-enabling techniques to realize this vision.
In this paper, we focus on applying SDN principles to a transport network research platform integrating backhaul and fronthaul (crosshauling) tra c, which features heterogeneous wireless links, namely mmwave and Wi-Fi [10]. The application of such principles allows a uni ed and dynamic control and management of integrated 5G multi-technology wireless transport networks [4]. This contrasts with cumbersome, error-prone, and ine cient traditional network management approaches based on the con guration of proprietary devices [15], which is often done manually.
In particular, our contributions in this paper are as follows. First, we detail the design and implementation of the wireless interface agent (WIA) integrated in our SDN research platform [10]. Its purpose is to manage the wireless transport interfaces of its nodes. The WIA can manage heterogeneous transmission technologies, e.g., mmwave and Wi-Fi, and it is employed to characterize the performance of the deployed mmwave network. Second, we measure the downtime of the mmwave network in the presence of link up/down events triggered with the WIA, with obtained response times in the order of 10s-to-100s of milliseconds depending on the case. Third, the TCP performance over the multi-hop mmwave network is also experimentally evaluated. In fact, TCP throughput up to around 800Mbps are obtained for single and multi-hop scenarios despite neighboring links using the same channel. Finally, one can also observe the impact of Maximum Transmission Unit (MTU) size on TCP throughput, which may hinder the full exploitation of the mmwave link capacity when combined with other transport technologies, since the advantages of big MTUs (much bigger than the typical 1500 bytes) o ered by mmwave devices may not be reaped.
The remainder of this paper is organized as follows. We summarize related work in Section 2 with respect to management of wireless SDN transport networks and characterization of experimental mmwave networks. Section 3 presents the developed agent to manage mmwave devices in an SDNenabled wireless mesh network. Section 4 and Section 5 present the scenario under assesment and its characterization, respectively, which has been obtained using the presented management agent. Finally, we conclude our work in Section 6.

BACKGROUND AND RELATED WORK 2.1 Management agents for wireless SDN networks
Currently, the control and the management of wireless transport equipment is mostly based on proprietary solutions using di erent versions of Simple Network Management Protocol (SNMP) protocol or cumbersome command line interface (CLI), limiting automation and requiring specialized network engineers [15]. In order to overcome these limitations, the Open Networking Foundation (ONF) created the Wireless Transport SDN group to apply SDN principles to wireless transport networks. In their rst Proof of Concept (PoC), back in 2015, they considered extensions of the OpenFlow (OF) protocol to handle not only the control but also the management of wireless transport equipment. This approach has also been followed by other research work, such as [3]. However, the Wireless Transport SDN group of the ONF realised that common information models of network devices are required in order to achieve a more automated and uni ed management of wireless devices, decoupling the management from the control operations (forwarding behavior).
Then, the following PoCs of the Wireless Transport SDN group of ONF (the last one carried out in August 2017 [8]), demonstrated the capabilities and bene ts from utilizing a common Information Model (ONF TR-532) for multi-vendor management of microwave transport network devices. This information model is exchanged by means of the Network Con guration (NETCONF) protocol, a general-purpose management protocol, which uses YANG modeling language to implement the data model. At the network device, an adapter, what ONF calls mediator, is used for translating the NETCONF/YANG information model to/from the existing proprietary management procedures of each vendor's device. This approach is similar to the one we present in Section 3. However, our approach aims to manage devices using di erent technologies, namely mmwave and Wi-Fi, and we use RESTConf interface to exchange the information, allowing a simpli ed transaction model with respect to NETCONF, hence allowing for faster prototyping.

Characterization of experimental mmwave networks in the 60GHz band
The interest of mmwave technology to enable 5G network system design is evident due to the availability of large contiguous spectrum availability to deliver high speed data rates. Between the mmwave bands, the V-Band (60GHz frequency band) is of special interest because it is a worlwide licence free spectrum. This band is used in the IEEE802.11ad/WiGig standard [9], a promising solution for mmwave radio communications, which could be used for mobile crosshauling purposes in 5G ultra dense networks. However, there is still a shortage of experimental evaluation of IEEE 802.11ad networks. Some examples on the characterization of real singlehop IEEE 802.11ad mmwave networks are [6], [11] and [12], which consider di erent scenarios and conditions such as outdoor picocells, indoor o ce environment or the e ect of co-channel and adjacent channel interference. However, these works focus on single-hop link-level behavior characterization and do not consider interface management of mmwave nodes in a SDN scenario. Closer to our work, we nd [5], where resiliency mechanisms based on the OF protocol in a multi-hop SDN-enabled IEEE 802.11ad mmwave mesh testbed are assessed with UDP ows. Contrarily, this paper focuses more on the management part of mmwave Paper Session 1 mmNets'18, October 29, 2018, New Delhi, India devices in an SDN-enabled network and how the developed agent is employed to characterize the performance of the deployed mmwave involving control and data plane SDN architectural layers. It is worth highlighting that for the latter, TCP ows are used, as opposed to UDP in previous work [5]. Figure 1 presents the architecture of WiseHAUL [10], an SDN-empowered platform combining heterogeneous wireless technologies (mmwave and Wi-Fi) to conduct research and experimentation on SDN/NFV for 5G mobile transport networks. This section deals with the Wireless CrossHaul Forwarding Elements (WXFE) present in the WiseHAUL platform, focusing on the developed Wireless Interface Agent (WIA). As we can see in Figure 1, the SDN agent embedded in the WXFE has two components. This separation obeys to the logic of decoupling the control operations (forwarding behavior) from the management operations (con guration and administration of network elements). On one hand, we have the Forwarding Agent (FA), which is the OF protocol [7] agent provided by the instance of the open source software switch running in the transport node. This agent oversees forwarding actions and it is technology-agnostic of the underlying wireless transport technology (e.g., mmwave, Wi-Fi) since the matching headers taken into account to establish ow rules present an equivalent frame format. On the other hand, each wireless transport node counts with its own WIA, which exposes an Application Programming Interface (API) to the WiseHAUL SDN controller and upper application layers allowing them to con gure the underlying wireless network devices. The WIA is able to integrate interfaces of heterogeneous wireless technologies, by means of an appropriate information model de ned using the YANG modeling language. Then, the WIA is in charge of mapping the request to the speci c con guration command of each interface of the transport node. The WiseHAUL SDN controller and the WXFE communicate via a RESTConf interface over HTTP requests, where a uniform resource identi er (URI) identi es every exchanged resource. The client runs in the WiseHAUL SDN controller and the server runs in each WXFE.

WIRELESS INTERFACE AGENT FOR SDN WIRELESS DEVICES
This WIA approach (YANG-RESTConf) provides exibility and scalability capabilities to the solution if we compare to custom extensions of OF protocol, since the addition of new features is handled easily by extending the information model and de ning its associated URIs. An extension of OF protocol, initially proposed in the 1st PoC of the Wireless Transport Group of ONF and in [3], requires the de nition of new structures and parameters within the OF protocol structures/messages and the corresponding inclusion of code into the internals of the used open source software switch and the SDN controller to appropriately parse new de ned OF messages, hence limiting the scope and scalability of the solution. Furthermore, the use of RESTConf over NETCONF simpli es the interface, easing the prototyping, since the set of NETCONF operations can be implemented with the reduced set of HTTP methods.
In addition to con guration parameters, the WIA considers asynchronous noti cations of network events following a publish/subscribe pattern, which could be used to trigger actions at the SDN controller level based on monitored events at the network nodes.  Table 1 summarizes the set of currently supported parameters by the WIA, based on capabilities of available hardware using Wi-Fi (Compex WLE900VX IEEE 802.11ac) and mmwave (TensorCom TC60G-USB3.0 EVK IEEE 802.11ad) technologies. The available mmwave devices count with reduced con guration capabilities with respect to the considered Wi-Fi device because it is a pre-commercial device. In particular, current capabilities of the mmwave device drivers are reduced to read/con gure its modulation and coding scheme (MCS) and link status, and read its central frequency.

EXPERIMENTAL SETUP
We have con gured a four mmwave node transport network topology using ve nodes of the WiseHAUL platform, where four nodes are WXFEs and the remaining one is running the Paper Session 1 mmNets'18, October 29, 2018, New Delhi, India WiseHAUL SDN controller. The used setup is depicted in Figure 2. The distance between N4-N2 and N5-N3 WXFEs is four metres, while the distance between N4-N5 and N2-N3 WXFEs is two metres. Each node is based on an Intel Core i7 processor (12-Cores at 3.3GHz x86 CPU) with 32GB and 1TB of hard disk, running the Ubuntu 16.04 LTS distribution. Each node is equipped with three Compex WLE900VX IEEE 802.11ac Wi-Fi cards running the ath10k driver and the rmware provided by Candela Technologies [1], which is necessary to establish IBSS connections with adjacent nodes. They also count with two Gigabit Ethernet ports, one dedicated to testbed management purposes and the other one to set an out-of-band wired SDN control plane channel. WXFE has also USB 3.0 slots where up to two TensorCom TC60G-USB3.0 EVK devices are plugged to provide the platform with IEEE 802.11ad mmwave capabilities. The mmwave devices are mounted as network interfaces when loading its accompanying drivers in the kernel. In order to establish the mmwave link, the Ten-sorCom devices are con gured to operate as PBSS Control Point (PCP) mode on one side and as station on the other side. This mode con guration is done by means of control scripts also provided by TensorCom Inc. WXFEs use Open-vSwitch (OVS) with version 2.7.0 as software switch. In these switches, the di erent wireless interfaces, both mmwave and Wi-Fi, are added as OVS ports for transport purposes. An additional port using linux namespaces is added to the OVS switch to generate/receive tra c.

EXPERIMENTAL RESULTS
In this section, we present the performance characterization of the described experimental setup. This is done through two case studies. The rst case study provides results showing the experienced downtime after a change in the status of the mmwave link, while the second one studies the experienced TCP throughput in single and multi-hop mmwave setups when varying MCS and MTU. The experiments are executed with the use of the described WIA, triggered from the Graphical User Interface (GUI) of the WiseHAUL SDN controller. The reported values represented in the following gures show the statistical distribution of the corresponding experiment from the 25 th to the 75 th percentiles, and the whiskers from the 5 th to the 95 th percentiles. The marker represents the average value.

Case Study I: Setting the link status
The following results show the experienced downtime in a mmwave link when there is a change in its status. This change can happen unintentionally, due to a failure in the device, or intentionally through a management decision.
This study is possible thanks to the recovery possibilities provided by the WiseHAUL SDN controller [4] to react in front of network events, such as a change in the network topology. After a link event (either down or up) is reported, the WiseHAUL SDN controller checks the a ected paths, computes, deletes and reinstalls alternative paths (if possible). In this case, a "break-before-make" strategy is implemented. Next, we compare the downtime of a mmwave link with the one experienced by the Wi-Fi IEEE 802.11ac link. In this way, we show how the WiseHAUL SDN controller interacts with the WIA and the FA of the di erent WXFE depicted in Figure 1 and the capabilities of the WIA to simultaneously manage heterogeneous wireless technologies.
In order to characterize this downtime, we de ne the following experiment. We use the Iperf [14] tool to generate a UDP ow between the endpoints present at nodes labeled as N5 and N3 in Figure 2. After an instant, we switch o the mmwave interface of N5 connecting with N3 using the WIA. This action generates an OFPT_ PORT_STATUS message at N5, which triggers the recovery procedure at the WiseHAUL SDN controller. Then, the WiseHAUL SDN controller recon gures the FA of the di erent nodes with the appropriate OFPT_FLOW_MOD rules to resume the a ected ow through the alternative path composed by nodes N5-N4-N2-N3. After some instant, we use the WIA to switch on again the mmwave interface of node N5. A new OFPT_PORT_STATUS message is generated. This message triggers the recovery procedure, which reestablishes the former path, i.e, N5-N3. We de ne the downtime as the time interval when there has been an interruption in the reception of data packets at the endpoint of node N3 as a consequence of the switch o /on action. We repeat the same experiment 20 times with a mmwave link and a Wi-Fi link at the 5GHz band. Experiments were performed at night to reduce possible interference e ects in the Wi-Fi link. Figure 3 presents the experienced downtime when considering the di erent transitions and the di erent available wireless technologies. Figure 3 shows that the mean measured downtime after a switch OFF event is of 150ms for the mwWave interface, and slightly higher for the Wi-Fi  interface, around 180ms. The packet reception resumes as soon as the FAs install the associated ow rules at the corresponding WXFEs, as all the remaining interfaces in the alternative path are in ON state. However, the case for the OFF/ON transition is di erent. As we can see in Figure 3, the "mmwave downtime" is less than half the value (around 56 ms on average) with respect to the ON/OFF transition, and it is much lower than the Wi-Fi case, which is in the order of seconds. From the point of view of control operations requested by the WiseHAUL SDN controller to recon gure the forwarding behavior of the WXFEs, a lower downtime for the OFF/ON is expected because the recovery procedure implies more ow deletion operations in the di erent nodes (N5-N4-N2-N3) and less creation operations (N5-N3), which are more time consuming. In this case, the di erence in experienced downtime comes due to the time it takes to the wireless device to make e ective the order sent by the WIA. The mmwave link is established automatically after the interface is ON again. However, in the Wi-Fi case, it is required to establish again the IBSS network forming the link. Since the information of previous link (frequency, bssid) is lost after the OFF operation, the creation of the new Wi-Fi link incurs bigger delays, around 5.2s on average. In the best cases, this delay is of around two seconds, however, in the worst cases, it could arrive to more than eight seconds.

Case Study II: Setting di erent Modulation and Coding Scheme
The following results show a TCP throughput characterization of the deployed mmwave network when setting the di erent available MCSs with the WIA under single and multi-hop communications and considering di erent maximum transmission units (MTUs). Di erently to [5], we consider TCP connections between endpoints. TCP is selected given its signi cant share on Internet tra c. Additionally, this evaluation gains relevance for our setup, since the performance of TCP connections can be substantially a ected due to the nature of mmwave communications, i.e, high data rates transmitted over the di erent links of a multi-hop communication with potential channel variations. The evaluated ows last for ten seconds in each repetition, and are generated using the Iperf [14] tool. Every conducted experiment is repeated 10 times. For the case of one hop, TCP tra c has been generated between the endpoints of the nodes labelled as N2 and N3. For the case of two hops, tra c has been generated between the endpoints of the nodes labelled as N2 and N5, using N3 as forwarding node. TensorCom TC60G-USB3.0 EVK devices at N3 are con gured as PCPs while devices at N2 and N5 are con gured as stations. Table 2 present round-trip time (RTT) measurements between considered endpoints. These measurements have been obtained with 100 samples over 100 seconds.
Additionally, this characterization also explores the e ect of the MTU on the throughput network performance. The maximum value of supported MTU of TC60G-USB3.0 EVK devices is 7912 bytes. However, we also consider 1500 bytes, which is popularly used in Wi-Fi networks and in Internet. As commented before, WiseHAUL platform combine di erent wireless technologies. This heterogeneity will be the norm in 5G network deployments and it is interesting to see how this can impact the performance of the network, since the full capabilities of mmwave hardware would not always be possible to be exploited. In our setup, we have considered that all mmwave links operate in channel 2 (60.48GHz), di erently to [5], which considered di erent channels for the di erent mmwave links. As a member of the IEEE 802.11 family, the multi-hop communication in the presented setup is likely to be a ected by co-channel interference (CCI), because both links share the same channel without being aware that they are in each other carrier sensing range. In addition to this, the CCI is also likely to occur due to the nature of the mmwave equipment under evaluation. As explained in [13], pencil-shape beams in mmwave networks are a myth. The reason for this is that real-world phased array antennas su er from signi cant side lobes, and its e ect is particularly noticeable in consumergrade hardware due to its cost-e cient design. Finally, it is worth mentioning that the IEEE 802.11ad standard only de nes four channels, reducing the possibilities at designing the frequency planning and making co-channel interference more likely for close devices. Figure 4 shows the performance comparison of end-toend application throughput when considering the variables previously mentioned: available MCS, number of hops and di erent MTU values. As expected, the maximum throughput has been achieved using MTU 7912 bytes with one hop and using the fastest available MCS (MCS7). This value is of  [13], the cause of a good performance at the receiver of the second hop is not only explained due to the selection of a good beam pattern capturing as most as possible of the main lobe of the desired transmitter but also due to very residual reception of side lobes transmissions from the other interfering transmitter.
If we refer to the values obtained with MTU 1500 bytes, we can see that using higher MCS schemes does not provide further gains in terms of end-to-end throughput and there is a saturation as depicted in Figure 4, and also experienced with TCP connections in [12], but for the case of one hop. In such cases, having network elements with a value of MTU equal to 1500 bytes, it could be convenient to use a lower MCS in mmwave devices (giving a similar performance than higher MCS) because they provide more reliable communications requiring less sensitivity at the receiver side, and at the same time, providing energy consumption gains due to reduced processing costs at the transmitter/receiver units.

CONCLUSIONS
In this paper, we present an agent (WIA) to manage the wireless interfaces of a real SDN-based multi-hop transport network research platform featuring mmwave and Wi-Fi communications. The experimental evaluations carried out suggest that such a RESTConf-based management agent is fully functional and enables fast prototyping, as opposed to more cumbersome options (e.g., modi cation of OF protocol). We employ the WIA to perform a quantitative evaluation of the deployed mmwave network, where downtimes in the order of 10s-to-100s of millisecond can be obtained in the presence of link up/down events. Furthermore, the maximum TCP throughput can be achieved in mmwave multi-hop scenarios despite neighboring links being con gured in the same channel and using cost-e ective pre-commercial equipment. Another relevant conclusion is that MTU mismatch among technologies in heterogeneous networks may hinder the full exploitation of the capacity o ered by mmwave links, which achieves its maximum throughput with big MTUs.