Putting NFV into Reality: Physical Smart Manufacturing Testbed

5G and network function virtualization (NFV) have the potential to make factory networks more flexible and scalable. For this purpose, we defined use cases in previous work and demonstrated them using emulation-based prototype deployments. In this paper, we extend our previous work and deploy our developed vertical-specific network services and cloud-native virtualized network functions (VNFs) in a physical NFV testbed that is located on the shop floor to analyze the potential of NFV in real-life environments. We use the 5GTANGO service platform to deploy them on Kubernetes. Both are running on top of OpenStack.


I. INTRODUCTION
In recent years, huge effort was invested in the fifth generation of mobile networks 5G. This concerns standardization, research on new technologies and other innovation actions. For the vertical industries, 5G and the associated technologies are of great interest. The 5G Infrastructure Public Private Partnership (5G PPP), which is a joint initiative between the European Commission and European ICT industry, has identified the "factories of the future" as one of the most important vertical sectors for 5G [1], [2]. In their white paper [3], 5G PPP presented use case families and representative scenarios in manufacturing and highlighted their potential impact on manufacturing. Furthermore, technical requirements from manufacturing for 5G were addressed; e.g., timing requirements, heterogeneity requirements, security and safety requirements, network infrastructure requirements and service management requirements. In order to achieve these, changes to existing technologies are necessary which leads to the development of next generation radio technologies, such as Non-Orthogonal Multiple Access (NOMA) schemes [4] to service-level technologies, such as hybrid clouds [5], as well as backhaul and core network technologies, such as software defined networking (SDN) and network function virtualization (NFV).
Weidmüller, a large-scale manufacturer that supports customers and partners around the world with products, solutions and services in the industrial environment of power, signal and data, has already identified the potential of NFV early. Together with other partners Weidmüller is doing research on the application of NFV in manufacturing which takes place in the context of the EU-funded 5G PPP research project 5GTANGO that is explained in more detail later in this paper. NFV is an emerging technology that promise more flexible and scalable networks, which is of great interest in manufacturing After reviewing related work in Section II, we discuss the integration of NFV into factory networks in Section III considering advantages and challenges. In Section IV, we focus particularly on the cooperation between NFV and IIoT, and in Section V we present our NFV testbed as an approach for integrating NFV into factory networks. Finally, we conclude our paper in Section VI and give an outlook on future work.

II. RELATED WORK
Wollschlaeger et al. reviewed technology trends in industrial communication and identified 5G as a potential disruptive technology, but also discussed the challenge of managing complexity and heterogeneity through NFV. In this paper, we tackle the challenge putting NFV into a physical smart manufacturing testbed for enabling machine data aggregation, but do not consider SDN in this scenario. Kálmán et al. [7] and Ahmed et al. [8], proposed SDN for industrial automation networks. Silva et al. extended the OpenFlow protocol in [9] to manage real-time reservations provided by certain switches in the data-plane and assessed SDN and time sensitive networking (TSN) in [10] and compared how efficiently TSN and SDN address the requirements of Industry 4.0. Their results indicate that SDN should be examined more closely in future work. This could also be done with a focus on industrial automation network protocols based on TCP/IP suite, such as ModbusTCP [11] or EtherNet/IP [12].
Nevertheless, Behnke et al. proposed already the concept of a business process logic controller (BPLC) in [13] to close the gap between business processes and network management by enabling a direct network control for business process systems, such as enterprise resource planning (ERP) systems and manufacturing execution system (MES), using SDN and NFV. These systems have a full overview of all ongoing processes and their knowledge can be used for fast reactions on the network topology.
Furthermore, Behnke et al. [14], proposed a first concept for a smart manufacturing use case that is focused on an NFVbased machine interconnection network service. Requirements were addressed and useful general purpose virtual network functions (VNFs), such as a firewall and an intrusion detection system (IDS), were identified.
Schneider et al. [15] and Müller et al. [16] built on this concept and enriched the machine connecting use case with more details. They designed manufacturing-specific VNFs that are necessary to realize the use case and composed them into two network services. The machine interconnection network service (NS2) contains amongst other VNFs a machine data collector (MDC) VNF that collects data from machines and forwards them to the factory edge network service (NS1). NS1 contains a cloud connector (CC) VNF that makes it possible to publish this data to a factory cloud backend, such as Microsoft Azure, and provide it also for other applications via its integrated MQTT broker: Eclipse Mosquitto 1 .
In addition to both network services, a third network service was proposed in [13] and [16] for enabling augmented reality (AR) applications on-demand; e.g., for remote maintenance of an injection molding machine using Microsoft HoloLens 2 ( Figure 1) or a maintenance assistance system.
Peuster et al. [17] used the factory edge network service (NS1) and the machine interconnection network service (NS2) and demonstrated their application on an emulation-based prototyping platform for NFV [18]. They used an injection molding machine simulator software (IMMS) to simulate machine data, e.g., a part counter, the actual cycle time, etc., as well as a free manipulable constant numerical value, and a configurable sinusoid. An instance of the IMMS offer this data via Euromap 63 [19], a file-based data exchange standard for injection molding machines. The network services are used to transport this data into Microsoft Azure. Finally, Microsoft Power BI 3 was used to visualize this data and plot the defined sinusoid. The implemented prototype with all custom-built VNFs is available as an open source project [20].
In this paper, we go beyond an emulation-based prototyping platform for NFV. We deploy network services and their VNFs in a physical smart manufacturing testbed and use these as components of our factory network.

III. INTEGRATING NFV INTO FACTORY NETWORKS
The information technology (IT) has been supporting manufacturing for a long time. Corporate IT systems, such as enterprise resource planning (ERP) and product lifecycle management (PLM) systems, provide master and product data, and offer tools for business process and production planning. Usually, these systems are maintained by the corporate IT, while production machines and their networks are maintained by machine maintenance departments. This means, IT and operational technology (OT) are usually logically separated, but due to the ongoing digitization, e.g., implementation of manufacturing execution systems (MES), usage of industrial IoT and mobile devices, IT and OT converge. The convergence of IT and OT is a challenge, but it is unavoidable and a necessary step into the future of modern factories. The company and its technical staffs will profit from the convergence. The staffs learn from each other, exchange information, share knowledge and transport ideas and concepts. Furthermore, they can collaborate on exploring, testing and integrating new technologies when they arise. The collaboration reduces time and costs and facilitates the fast integration of new technologies in both worlds, IT and OT. As a result, NFV will find the way also into factory networks.

A. Benefits of NFV-based factory networks
Factory networks can profit from NFV because it makes networks more flexible; network services can quickly be instantiated and provide general purpose network functions and manufacturing-specific services on-demand; and they can be terminated when they are not longer needed without touching the hardware. This can reduce costs and time of network reconfiguration. For example, machines are frequently moved on the shop floor due to optimization and reorganization of production lines which can necessitate network reconfiguration.
Furthermore, NFV makes networks more scalable. The number of devices communicating in factories increases due to the ongoing digitization of manufacturing which increases the network traffic. In addition to the increasing network traffic, the load is not constant due to the application of mobile devices, such as smart phones, tablets and smart glasses. This means, quick and maybe, temporary adjustments of the network are necessary; e.g., considering the concept of a load balancer, a network service can automatically be instantiated on-demand and terminated when it is not longer needed. Such softwarized and programmable network components increase the automation of network control and enable autonomous configuration of the network which can reduce time and costs of flexible factory networks.
In addition, dealing with NFV entails dealing with virtualization tools and technologies. As a result, further domains become visible that can profit from virtualization than networks. For example, OpenStack 4 and Kubernetes 5 can also be used to deploy standardized services in the factory network; for example a database. This makes it possible to react quickly on demands. Furthermore, standardized images/containers are deployed which reduces effort in supporting these systems.

B. Challenges of integrating NFV-based factory networks
A first challenge has been mentioned before. When IT and OT converge, tasks will overlap and technical staffs from both departments have to collaborate more closely. We assume, that in the near future, machines are still connected by technical staffs from OT departments. This means, they will need training, e.g., in designing network architectures and orchestrating NFV-enabled networks. Alternatively, NFV tools are necessary to create an abstraction layer to network services and VNFs, so that they can be used without deep knowledge in these techniques and concepts. This is addressed and explained in more detail in Section V-B.
Network services and their VNFs must also fulfill industryspecific requirements, e.g., availability, reliability, timing requirements, etc. (see also Section I). The fulfillment of these requirements can be checked by using appropriate test equipment such as the 5GTANGO verification and validation (V&V) platform (Section V-A). Nevertheless, an NFV-enabled infrastructure is necessary. This is described in detail in Section V.
NFV replaces network functions on dedicated appliances by virtual instances defined by software running on commercial of-the-shelf (COTS) hardware. For the application in manufacturing environments, industrial-grade hardware is necessary. Furthermore, the form factor must be considered because 19 inch racks are necessary for 19 inch hardware, but these are not everywhere available in factories. An alternative could be the application of DIN rail bare metal hardware for NFV.
In the next section, we are focused on correlations between NFV and IIoT.

IV. PAVING THE WAY FOR IIOT IN MANUFACTURING
In the next years, the number of IoT devices that are integrated on the shop floor increases.

A. NFV as enabler for factory network security
Many IoT devices use WiFi for communication. Potentially, it is the same WiFi infrastructure that is used for the production machines. This means, IoT devices can compromise the network security and make it insecure. For example, IoT devices can have security breaches, such as personal computers, but often, updates for IoT devices are not regularly provided or patches are not automatically installed.
This problem is well-known from computers of production machines. Hardware, operating system and machine control software are matched to each other, patching those systems frequently and switching to the newest version of the operating system is very cost-expensive or unfeasible. Therefore, these systems are encapsulated, sub networks and only well-defined interfaces are used. For example, a firewall is used to block undesired connections, so that communication is only possible with unblocked IP addresses through individual ports. Furthermore, IoT devices could also be insecure due to usage of untrustworthy hardware. Of course, the best strategy is to avoid the application of such devices, but this is hardly possible. The best alternative is to encapsulate devices and control the traffic. This concept prevents devices from being damaged and therefore excludes further damages on other devices.
NFV can support this concept through encapsulation and traffic control. For example, a pre-defined blocking VNF could be used to control traffic for known device types, while unknown device types are initially isolated.
In addition, NFV can also be used to complement monitoring of the network using an intrusion detection system (IDS). Due to the NFV-enabled infrastructure, devices can automatically be isolated without any human interaction if a threat is detected.
This means, NFV paves the way for IoT devices in manufacturing, and it is an enabler for factory network security. Nevertheless, NFV can also profit from IoT devices. For example, industrial IoT devices can be used as actuator to control physical network infrastructure.

B. IIoT as actuator for network control
Today, standards such as IEEE 802.3 Ethernet and IEEE 802.11 Wireless LAN (WLAN, WiFi) are established and used on the shop floor. Nevertheless, used principles like listen-before-talk exacerbate the usage of a massive amount of devices. Therefore, factories are very interested in mobile broadband cellular network technologies like 5G. WiFi is wellestablished in offices and also on the shop floor. There is a large number of user equipment available on the market and a large number has already been rolled out on the shop floor. Therefore, WiFi will also be present on the shop floor for the next few years and must be considered.
Two frequency bands are used for WiFi technology: 2.4GHz and 5GHz, and each have their own set of channels, which are a resource. Their usage should be managed to reduce interference, such as co-channel interference (CCI) and adjacent channel interference (ACI) as well as interference with non-WiFi devices that also compete for medium access. In Fig. 2: Demonstration of using Microsoft Power BI for data visualization on Microsoft HoloLens [13], [16] augmented reality on-demand was proposed. In the scenario addressed in both papers, the Microsoft HoloLens is used by a technical staff to visualize machine data. This could be done by dashboards created in Microsoft Power BI. Machine data is transported using the network services NS1 and NS2. Figure 2 illustrates the application of Microsoft Power BI with Microsoft HoloLens.
Müller et al. propose in [16] a third network service (NS3) that contains a router which interconnects the local AP with the Internet and enables the Internet connection of the Microsoft HoloLens on-demand. This concept can be used to reduce interferences due to competing WiFi devices (APs) by power off APs that are not needed. Industrial WLAN APs could be powered on and off using an IoT controller, such as Weidmüller's UC20-WL2000-IOT 6 , that is equipped with a digital output module and interconnected with the network. A control application can be programmed using JavaScript by Node-RED to set a digital output signal to power on the Industrial WLAN AP on-demand, and power it off when it is not longer needed. The application subscribes a defined topic from a defined broker that represents the status set point of the defined WLAN AP (on or off). For this purpose, an own MQTT broker could be instantiated, such as Mosquitto. Usually, the topic is set to off, but when the WiFi AP is needed the topic is switched to on. This could be done by human interaction in a Factory Management Portal (FMP), that is proposed in Section V-B, or due to decisions made autonomously by the NFV-enabled platform; for example, a monitoring system has detected the demand. This was an example that demonstrates how NFV and IIoT can cooperate.

V. NFV TESTBED
Weidmüller integrates the NFV testbed in its industrial IoT testbed, which is located on Weidmüller's premises in Detmold (Germany), so that interaction with physical injection molding machines and further industrial components, such as IoT devices, become possible. However, the first tests are done using an injection molding machine simulator. In the next sections, we describe the testbed and its infrastructure in detail.

A. Components of the NFV Infrastructure
As proposed in [16], we planed to migrate the deployment from NFV-enabled emulator to an NFV infrastructure using Kubernetes. Traditionally, virtual machines (VMs) are used with guest operating systems that run on top of a hypervisor, such as OpenStack, but we use Kubernetes for deployment of container based network services. Containers are virtualized in the operating system; they have a smaller footprint, need less processing resources and can be deployed quickly. We use Docker 7 to build lightweight containers for our VNFs.
In particular, we use the SONATA SP powered by 5GTANGO, which is also named 5GTANGO SP, to deploy our developed network services on Kubernetes. 5GTANGO 6 Weidmüller IoT controller: https://www.weidmueller.com/int/products/automation software/controls/ 7 Docker: https://docker.com/ [21] is a platform that puts forth the flexible programmability of 5G networks and deals with NFV technologies. The NFVenabled platform consists among others of: • a service development kit (SDK) [22] with VIM emulator [23] for developing and testing network services, • a service platform (SP) as management and orchestration (MANO) platform [24], • and a verification and validation (V&V) platform [25] for testing network services on the appropriate target service platform. As proposed in [22] alternative systems, such as Open Source MANO (OSM 8 ), can also be used, but this is beyond the scope of this paper.
The 5GTANGO platform is controllable using a web-based graphical user interface or a command-line interface, but the direct usage of both interfaces requires deep knowledge about the network services and their life cycle management. Therefore, we use a factory management portal (FMP) as an abstraction layer to increase the acceptance for NFV in factory networks and reduce entry barriers. The factory management portal (FMP) was already mentioned in Section III-B and proposed in [15], [16]. It is an abstraction layer for technical staffs. The FMP makes it possible to work with NFV-enabled infrastructure and instantiate network services without deep knowledge about it. Figure 3 shows machines managed by the FMP. Ten machines are connected, three machines are disconnected, and one is moved from right to left via drag and drop. For each machine, a machine interconnection network service (NS2) is instantiated. It is managed and orchestrated by the 5GTANGO SP, but the FMP prevents technical staff from working with the SP directly. They do not have to worry about network service instantiation, life cycle management, performance, or IP address management. The status of each machine corresponds to the network service status; i.e., a machine is connected when its network service is running, it is disconnected, if it was terminated, and it is isolated if the isolate flag was set and the network service is not connected to the factory edge network service (NS1). The FMP is designed for NFVenabled network management of production machines, such as injection molding machines, but in the future, managing IoT devices will also be considered.

B. Factory Management Portal (FMP)
C. Architecture of the NFV Infrastructure Figure 4 shows the architecture of the NFV infrastructure applied on Weidmüller's premises. COTS server hardware is used: HP ProLiant DL380 Gen10 with Intel Xeon Gold 5118 CPU @2.30GHz, 2 processors (12 cores), 64 GB RAM, and 400 GB SSD storage.
OpenStack is used for VM management. The 5GATNGO service platform (SP), the FMP and Kubernetes are installed on separated VMs. The FMP instructs the SP to deploy a particular network service. The SP handles the automatic creation of Kubernetes pods and services to facilitate the deployment and interconnection of the described network services. This is illustrated in Figure 4 by the blue arrows.
The factory edge network service (NS1) instantiated consists of two VNFs: CC VNF and EAE VNF. The EAE VNF is the edge analytics engine and consists of one cloud-native deployment unit (CDU): a Grafana 9 Docker container. The CC VNF consists of four CDUs: • CDU 1: Eclipse Mosquitto MQTT broker • CDU 2: Time-series database Prometheus 10 • CDU 3: MQTT Exporter (custom script) that forwards data from Mosquitto to Prometheus • CDU 4: Processor (custom script) that forwards data from Mosquitto to Microsoft Azure IoTHub 11 . The machine interconnection network service (NS2) consists of one VNF: machine data collector MDC VNF. It contains one CDU (custom script) and mount a samba shared folder (variant for file-based data exchange for Euromap 63).
Considering 99% confidence level, the average time to deploy the factory edge network service (NS1) on our system is 13.76s ± 0.84s. The average time to deploy the machine interconnection network service (NS2) is 7.42s ± 0.54s. NS1 has a higher instantiation time than NS2 because it involves more containers that need to be started. In [15] instantiation times were presented for using the NFV-enabled emulator. The instantiation time using 5GTANGO SP and Kubernetes is higher than on the NFV emulator, but this was expected due to using more complex platforms. Nevertheless, we assume that the instantiation time is significant lower than using heavyweight VMs instead of containers.

VI. CONCLUSION & OUTLOOK
In this paper, we discussed the integration of NFV into factory networks. In doing so, we have identified advantages and challenges, and we proposed concepts for approaching these. We presented an NFV testbed integrated in an industrial 9 Grafana: https://grafana.com/ 10 Prometheus database: https://prometheus.io/ 11 Microsoft Azure IoTHub: https://docs.microsoft.com/en-us/azure/iot-hub/about-iot-hub IoT testbed to demonstrate the application of NFV in factory networks. The result of the work presented is an implemented architecture for putting NFV into factory networks, so that the shop floor can profit from virtualization. As a next step, we need to gain experience with the implemented architecture. Future work will be on further development of network services; e.g., a VNF for intrusion detection could be used to increase the network security. Further future work will be on automated verification and validation solutions to ensure the correct behavior of the developed network services.