REALIZING THE NETWORK SERVICE FEDERATION VISION Enabling Automated Multidomain Orchestration of Network Services

The 5G-Transformer (5GT) project proposes a network function virtualization (NFV)-/software-defined networking (SDN)-based architecture to manage the end-to-end (E2E) deployment of composite NFV network services (NSs), which may involve multiple administrative domains (ADs), and hence require NS federation (NSF) capabilities. At the architectural level, this article presents the service federation functionality of the 5GT service orchestrator (5GT-SO). It covers the gaps identified in European Telecommunications Standards Institute (ETSI) NFV reports and specifications [e.g., Interfaces and Architecture (IFA) 028]. Some recommendations are also presented based on this experience, particularly on the relevance of multidomain resource orchestration. Experimental results show that the federated service under evaluation is deployed in under 5 min. The time profiling of the various processing federation-related operations shows its reduced impact in the experienced deployment time. A comparison of service deployments of increasing complexity also offers valuable insights.

architecture toward a network of services. Complexity, heterogeneity, and dynamicity will be the rule to serve the diverse requirements imposed by multiple vertical industries (e.g., automotive and eHealth) over the same shared infrastructure, using network slices, or spanning through different ADs. In this context, network automation and programmability become essential to support these objectives. It is widely accepted that SDN and NFV concepts represent the foundations to effectively provide these network capabilities.
To enable such a vision, the 5GT project [1] is developing an NFV/SDN 5G mobile transport network platform-based on ETSI NFV procedures, specifications, and reports-capable of managing the deployment of E2E NSs spanning complex and heterogenous transport networks using computing resources deployed in different points of presence (PoPs), and involving multiple ADs (each managed by a different 5GT platform instance).
We refer to the NS deployment in different ADs, each using its own management and orchestration (MANO) platform, as NSF. The 5GT platform can perform NSF thanks to its capabilities to orchestrate composite NS descriptors (NSDs) [2].
Composite NSs are made up of a set of constituentnested NSs. These different nested NSs may need to be instantiated in various ADs for a number of reasons, including a shortage of resources, NS availability, or the need to deploy an NS while satisfying certain requirements (e.g., the coverage of a given geographical area). Then, composite NSs allow for designing NSs not only as integral units that work on their own but also as sets of NSs that can be dynamically grouped to create more complex and tailored offerings to vertical industries. As a result, NSF can provide enlarged service coverage and connectivity, seamless service continuity, and flexibility to vertical industries. This enables the service providers (SPs) to offer a broader spectrum of NSs to vertical industries, opening the door to new business models. For instance, vertical industries may set up a business relationship with a single SP that satisfies its requests, while this SP establishes agreements with other SPs to extend its service offerings, thus increasing its profit by avoiding the rejection of service deployment requests.
In the literature, the NSF problem has been split into two parts. First, there is the problem of mapping the different parts of the NSs across multiple ADs. The work in [3] presents a solution based on integer linear programming and compares it with other previous solutions through simulation. However, these solutions do not delve into the required procedures and interfaces to truly deploy such NSs in different ADs, which is the second part of the problem and the focus of this article.
In that sense, different reports of standards development organizations like the Open Networking Foundation [4], the Metro Ethernet Forum [5], and the ETSI NFV [6] propose recommendations for procedures and interfaces to achieve NSF. The ETSI NFV work, which is the basis of the 5GT platform, goes further and defines an interface in the ETSI-NFV IFA030 specification [7] based on the ETSI-NFV IFA028 report [6]. The IFA030 specification presents a description of the interface between orchestrator platforms (OR-OR) to perform MANO operations among multiple ADs; this is the new interface reference point identified by IFA028. Basically, IFA030 considers using for the OR-OR interface a subset of the operations defined in ETSI NFV-IFA 013 [8] for the NSD and NS lifecycle management interface. The 5GT platform adopts these operations as a baseline to perform NSF; however, IFA030/013 are service-level interfaces (and do not handle resources).
The potential architectural options used to support NSF are reported at high levels only by IFA028, which does not properly address an essential aspect of NSF, namely, the interconnection of nested NSs running in different ADs. This is often a disregarded functionality that is fundamental to making E2E multidomain orchestration and network slicing a reality in practice. In fact, a slice is not only the logical construct defining an E2E virtual network but also the corresponding underlying resources assigned in exclusivity to a given tenant over a shared infrastructure [9]. We tackle this gap in this article and propose enhancements to the work of IFA028 by defining a more detailed workflow, including a set of additional interactions among the different SOs to achieve the interconnections of nested NSs at the resource level. To the best of our knowledge, this is the first work realizing and evaluating a complete NSF procedure, including the deployment of a composite NS involving different SPs in a real multi-PoP, multitechnology, and multi-AD experimental infrastructure.
The main contributions of this article are as follows: 1) providing a set of recommendations at the SO architectural level to translate high-level ETSI NFV report guidelines into a feasible operational implementation of NSF, including the two main aspects of E2E orchestration functionality (service and resource orchestration) 2) presenting the complete workflow that implements an NSF to explain what should be coordinated among peering domains at the service and resource levels 3) evaluating experimentally the automatic deployment of a composite NS in two ADs managed by different SPs, each running its own 5GT platform.

5GT Architecture Description
The 5GT architecture is based on the ETSI NFV work and aims at providing a platform with flexible and dynamic management capabilities to accommodate multiple and heterogeneous services derived from different vertical industries. This platform concurrently maps such services into a shared infrastructure combining multiple heterogeneous resources in terms of computing, storage, and networking. The source code of the 5GT platform is available in the 5GT repository [15] as open source code under the Apache 2.0 license. Figure 1 presents the main building blocks of the 5GT platform. It is worth mentioning that the 5GT platform  includes a certain level of security in its operations, although this is not its main scope. All of the blocks within the 5GT architecture keep a register of allowed associated blocks and requesters for each action to deter nonauthorized third-party entities from launching or modifying ongoing operations.
The 5GT-vertical slicer (5GT-VS) is the front end of the 5GT platform for vertical industries. It offers a high-level interface allowing users to easily request vertical services. The 5GT-VS focuses only on the service and business demands without taking over eventual service deployement at the resource level. The 5GT-VS presents a catalog of vertical services, particularized by the vertical users with their requirements. The internal logic of the 5GT-VS translates business-oriented service requirements into slicerelated requirements to manage the lifecycle of network slices, which are deployed as NFV-NSs.
The 5GT-SO oversees the E2E orchestration and lifecycle management of the NFV-NSs based on the available resources advertised by the underlying mobile transport and computing platform (5GT-MTP). The 5GT-SO embeds the network SO and the resource orchestrator (RO), whose functionalities are equivalent to those performed by typical NFV orchestrators [10].
In the 5GT, NFV-NSs may embrace one or multiple ADs; this means that the 5GT-SO, aside from interacting with its local 5GT-MTP (within a single AD), may interwork with other 5GT-SOs governing remote domains through the SO-SO interface to hence perform NSF.
The 5G-MTP is the unified controller responsible for orchestrating resources, virtualized network function (VNF) instantiation, and connectivity management at the underlying physical transport network. The 5G-MTP has full control of the underlying resources by interacting with different managers: the virtual infrastructure manager (VIM) for managing compute resources or the wide-area network (WAN) IM (WIM) for enabling network connectivity among different PoPs. For the sake of scalability and to simplify 5GT-SO operations, the 5GT-MTP applies abstraction mechanisms when exposing the resource view toward the 5GT-SO.
Finally, the 5GT monitoring (5GT-MON) platform provides metrics to the 5GT platform so it can perform reactive actions to continuously ensure targeted service-level agreements (SLAs). More specifically, the monitoring service at the 5GT-MTP collects data about the local physical and virtual resources, the 5GT-SO MON service collects data about the managed VNFs and NFV-NSs, and the 5GT-VS MON service collects data about network slices and vertical services. Figure 2 presents the internal software architecture of the 5GT-SO and its relationship with the 5GT-VS, 5GT-MTP, and 5GT-MON blocks. As explained in [11], the 5GT-SO introduces the concept of the service manager (SM), which allows for the exploitation of open source production-level MANO platforms [e.g., open source MANO (OSM) and Cloudify] by using wrappers. This approach permits focusing on innovative functionalities, such as the NSF, being easily supported regardless of the specific MANO platform.

5GT-SO Internal Software Architecture
The architecture of the 5GT-SO is organized in four main building blocks. The first is the SM, or the "brain" of the 5GT-SO. It embeds the service, resource, and monitoring orchestration logic, dispatching relevant tasks to the other building blocks according to the corresponding operational workflow. The second is the core MANO platform (i.e., OSM or Cloudify), which interacts with the 5GT-MTP to handle operations related to computing resources. The third building block is the placement algorithm (PA), which runs as a separate process of the 5GT-SO. The PA interacts with the RO module of the 5GT-SO through a well-defined REST application programming interface, enabling the exchange of PAs [12]. Finally, the databases maintain the state of the 5GT-SO system in terms of service offerings, instantiated NSs, 5GT-MTP used resources, and so on.

Network Service Federation Workflow
The 5GT-SO handles NSF in a way that is transparent to the 5GT-VS (and to the vertical user) thanks to two additional blocks introduced in the SM with respect to the baseline architecture presented in [11]. In the same way that any single-domain SO has two main functionalities (service and resource orchestration), a multidomaincapable SO, like the 5GT-SO, must introduce the corresponding service and resource orchestration entities needed for coordination with peering domains. These blocks (shown in Figure 2) are a hierarchical (parentchild) service orchestration engine (SOE) and a composite RO engine (CROOE). In the hierarchical SOE, the SOE parent (SOEp) focuses on the multidomain service orchestration aspects by coordinating with peering SOEps. In this sense, it analyzes the incoming NS requests and orchestrates the required operations at the different underlying submodules depending on such requests. It also coordinates the interaction with the resource orchestration block (CROOE), which handles the interconnections between nested NSs. In case of NSF, CROOEs of peering domains jointly handle the resources at both sides to establish the required connection. This is the fundamental step that enables a full E2E multidomain orchestration. The NSF workflow assumes that there was a previous "offline" agreement process between peering domains. In this offline process, SPs 1) establish physically the control and data plane links between ADs and update their 5GT platforms accordingly to incorporate information about federated domains and the available links and 2) onboard composite and nested NSs to the 5GT platforms of the different peering ADs, so it is known in advance in which AD a nested NS can be deployed. This workflow follows a similar approach to the top-down solution presented in the IFA028 report; however, this article enhances the IFA028 solution by explicitly dealing with the interconnection of nested NSs. Achieving this interconnection requires a set of new additional steps, where peering CROOEs exchange information related to instantiated VNFs through the defined SO-SO interface depicted in Figure 2. The specification of these exchanged messages is available in [15].
For a single NS, the SOEp relies on the SOE child (SOEc) to perform the NS instantiation operation following the original orchestration logic explained in [11] using ETSI NFV IFA 014, 011, and 013 specifications. For a composite NS, the set of operations depends on where the different nested services in a composite NS can be instantiated (either at the consumer or provider domain).
Initially, the SOEp decomposes the composite NS between nested NSs to be instantiated locally (consumer domain) and for nested NSs to be instantiated in a federated domain (provider domain). Then, the SOEp asks the CROOE block to determine how the different nested NSs in the composite NS are interconnected based on the NSD. Next, the SOEp starts the instantiation of the different locally nested NSs using the SOEc, as explained previously for a single NS. Unlike the workflow defined in IFA028, our proposal starts dealing with operations at the consumer domain because it considers this domain to be the coordinator of the process (as it is the receiver of the NS instantiation request).
When finishing the instantiation of locally nested NSs, the SOEp launches the instantiation of federated nested NSs, contacting the 5GT-SO of the corresponding provider domain through the northbound interface (NBI). At the federated domain, the NBI first validates that the request comes from an AD that is allowed to solicit NSF operation. Due to the presence of additional parameters in the instantiation request, the provider SOEp realizes that this request is from a consumer domain and asks this domain about the information required to coordinate the instantiation of the nested NS at the provider domain. This interaction between 5GT-SOs is done between its respective CROOE modules through the SO-SO interface. Then, the provider SOEp relies on its SOEc to instantiate the federated nested NS.
While the operation progresses, the consumer domain polls the provider domain periodically to check the instantiation status of the federated nested NS. Once the consumer domain has confirmed that the federated nested NS has been correctly instantiated, its CROOE module asks its counterpart at the provider domain about the instantiation details of the federated nested NS. This information is needed to set up the internested NS connections among the different VNFs and coordinate the instantiation of other possible subsequent federated nested NSs, while avoiding address collisions between ADs. After the different federated nested NSs have been instantiated, the SOEp manages the required interconnections between the different nested NSs, as expressed in the composite NSD.
First, it asks its CROOE to determine and set the interconnections among VNFs of different nested NSs deployed locally at the multiple PoPs under its control. Then, the CROOE block interacts with the local 5GT-MTP through the southbound interface (SBI) to set these connections.
Second, the SOEp asks the CROOE to interconnect VNFs of different nested NSs deployed in the PoPs of provider/federated domains with the ones instantiated at the local PoPs of the consumer domain. To do so, for each combination of instantiated consumer-provider nested NSs, peering CROOEs exchange the interconnectivity information (the addresses of pairs of VNFs at each AD requiring interconnection) through the SO-SO interface. Afterward, each CROOE contacts its 5GT-MTP to set their segment of the interdomain internested path, that is, from the different PoPs to the point in its transport network where the inter-AD link is.
Finally, it is worth mentioning that the design of the presented workflow considers the instantiation of composite NSs having more than two nested NSs and that the consumer domain could interact with more than one provider domain for the same composite NS. Thus, this approach is functionally more scalable than the approach proposed by IFA028 [6], which considers only a single provider domain and a composite NS constituted by two nested NSs.
In summary, based on this article, there are some key takeaways in terms of architectural design. First, a correct modular design allows for the adding of NSF/ multidomain functionality on top of already-available single-domain orchestration. Second, a multi-AD SO must handle both service and resource orchestration between peering domains; this includes introducing the corresponding entities in both domains and defining the corresponding interfaces and information models at both levels. Third, the service orchestration part requires minimum modifications compared to IFA013/030 for offering the NSF functionality. However, the fundamental component needed to offer full E2E control is the resource orchestration part, which exchanges relevant resource information between domains and interacts with the resource layer (5GT-MTP) to set up the interdomain interconnection according to the requirements and to avoid potential resource conflicts in the identifiers (IDs) of different domains (e.g., IP addresses). other through the SO-SO interface, which traverses over a virtual private network (VPN) connection. At the data plane level, the inter-AD links interconnect the transport networks of peering ADs. In this setup, there is a single inter-AD link implemented via a VPN connection, hence ensuring security in the communications between ADs.

5GT Experimental Setup
The following sections present the different deployed ADs following a bottom-up approach, that is, from the available physical resources to the MANO system. This experimental setup has been built to satisfy the requirements of the eHealth use case defined within the 5GT project.

Administrative Domain 1
At AD 1, two different PoPs are deployed and managed by respective OpenStack instances. Both PoPs provide the necessary storage, computing, and networking resources for the NSs to be deployed. PoPs are placed at both ends of the AD to emulate distributed computing resources at the different and remote segments of the transport network, i.e., the edge and core.
At the transport network level, AD 1 contains two different technological domains. At one domain, a ring of four forwarding elements (FEs) uses wireless millimeter-wave/ Wi-Fi (IEEE 802.11ad/802.11ac) links, representing the edge packet-switched domain of the transport network, as depicted in Figure 3. The other domain represents the core part, which relies on a multilayer network combining three packet switches with an optical wavelength-division multiplexing mesh-network counting having two colorless reconfigurable optical add/drop multiplexers and two optical cross-connect nodes.
This transport network is managed by a hierarchy of SDN controllers. Each of the aforementioned domains is controlled by its dedicated technology-aware SDN controller, acting as child controller. On top of them, a parent controller handles the E2E SDN transport orchestration. This parent controller is based on the Internet Engineering Task Force's application-based network operation architecture [14]. This architecture supports hierarchical deployments thanks to the use of a unified interface at both the SBI and NBI: the control orchestration protocol (COP). Finally, child controllers interact with the underlying FEs and are configured using the OpenFlow (OF) protocol in the wireless domain and in the packet part of the core domain. The optical infrastructure is controlled via an active stateful Path Computation Element using the Path Computation Element Protocol. A complete description of this multitechnological transport network and its control plane solution is available in [13].
The 5GT platform interacts with this infrastructure through the 5GT-MTP. The latter considers the parent controller as the WIM, while the different OpenStack instances are the VIMs. The 5GT-MTP is governed by the 5GT-SO, which, in turn, communicates with its associated 5GT-VS and the 5GT-SO of the peering AD 2. The MANO platform associated with the AD 1 5GT-SO is an instance of OSM Release 3.
Administrative Domain 2 AD 2, which represents another edge domain, is connected to AD 1 at the output of the core domain through the inter-AD link represented in Figure 3. In this case, the depicted ring of four available FEs uses Wi-Fi (IEEE 802.11ac) links. These FEs are configured using OF. AD 2 consists of a single PoP (S-PoP) managed by a different instance of Open-Stack. The transport network uses its own SDN controller, which also uses the COP protocol to handle connection requests. The SDN controller and the OpenStack instance are coordinated by the corresponding AD 2 5GT-MTP. Likewise, in AD 1, the 5GT-MTP is controlled by a 5GT-SO instance that relies on another instance of OSM Release 3 as the MANO platform. This 5GT-SO communicates with its own 5GT-VS and the peering 5GT-SO at AD 1.

5GT Experimental Results
We next report on an experimental analysis aimed at validating the NSF capabilities of the 5GT platform. To do so, we evaluate the service deployment time (SDT) using the setup described in the "5GT Experimental Setup" section. We define SDT as the elapsed time between the consumer 5GT-SO receiving the NS instantiation request and this 5GT-SO declaring the NS as being correctly instantiated.
Five different deployment schemes are evaluated, as summarized in Table 1. These experiments cover the full complexity range: the simple ones are the instantiation of a single NS, the intermediate ones are composite NS services (both S-PoP and M-PoP), and the complex ones are federated, involving multiple ADs. Each experiment is repeated 10 times.
As depicted in Figure 4, the composite NS under evaluation consists of two nested NSs. This composite NS was defined within the scope of the eHealth use case defined in the 5GT project. The first nested NS emulates a virtualized evolved packet core (vEPC) and consists of four VNFs, namely, Mobility Management Entity (MME), Home Subscriber Server (HSS), Serving Gateway (SGW), and Packet Data Network Gateway (PGW). The second nested NS is a monitoring back-end (MB) NS and consists of two VNFs, namely, a load balancer (LB) and processing server. an already-deployed nested NS (vEPC NS) in the case of an emergency, for instance] and where needed (using NSF to deploy in different ADs to meet certain service constraints). Figure 5 shows the statistical behavior of the SDT for each of the considered deployment schemes. The box stretches from the 20th and 80th percentiles, including the mean and the median values. The whiskers represent the maximum and minimum values.
As observed in Figure 5, an increase in the number of operations at each deployment scheme has an impact on the total SDT. The maximum measured SDT corresponds to the federated scheme (approximately 270 s), which is in line with the 5G target of achieving SDT in the order of minutes.
Looking more closely at the results, we can conclude that the most time-consuming operations are 1) the creation of virtual networks to support virtual links defined in the nested NSs and the instantiation of the VNFs (virtual machine booting up) (compare Nested-vEPC to Nested-MB in Figure 5) and 2) the setting of the required connections through the transport network (see the Composite M-PoP and Federated schemes in Figure 5 compared to the others).
It is worthwhile to note two more lessons from Figure 5. One is the reduced difference between the Composite S-PoP and Nested-vEPC schemes (roughly 13% on average), which is due to the difference in time experienced when deploying with VIM 2 (slower) in comparison to VIM 1. The other lesson is the decreased disparity between the Federated and Composite M-PoP schemes (around 5% on average), which is due to the proximity between ADs (both placed in the same laboratory premises) and, more importantly, the lower complexity of the AD 2 transport network and its managing entity (WIM 2) with respect to AD 1. Although it sets twice the number of connections in the Federated scheme with respect to the Composite M-PoP scheme (due to the involvement of both AD transport networks), the lower complexity of the AD 2 transport network keeps the experienced SDT in similar values. Figure 6 shows the statistical behavior (using the same representation as in Figure 5) of the elapsed time in the processing operations at the entities introduced    at the 5GT-SO to handle NSF operations and the interactions among ADs. Table 2 lists a description of such operations. Figure 6 reflects the time values experienced from the consumer domain perspective (i.e., the one driving all of the instantiation processes). As observed, these operations do not contribute significantly to the SDT attending to the time values presented in Figure 5, the processing carried out at the SOEp being the most significant component. However, as mentioned previously, the impact of all these interactions is low due to the proximity between the ADs. Nevertheless, with longer distances, e.g., one 5GT platform in Europe and another in Asia, the impact would be in the order of milliseconds, which does not have significant relevance in the final experienced SDT.

Conclusions and Future Work
The 5GT project designed an NFV-/SDN-based architecture to manage the deployment of composite NSs across multiple ADs in what is referred to as NSF. This article described the new entities and workflow introduced in the SO. The proposed enhancements cover the gaps identified in the work of ETSI-NFV [6], [7]. General recommendations on NSF architectural design were also provided based on experience: most notably, the importance of multidomain resource orchestration if full E2E connectivity is to be provided.
To the best of our knowledge, our solution is the first NFV NSF solution to be designed and implemented. Furthermore, it was evaluated using a real infrastructure with multiple technological domains. This evaluation compares various deployment schemes of increasing complexity (ranging from regular NS deployment to composite NS deployment and, finally, federation) to identify their impact on the SDT of associated operations. The main components contributing to the SDT are the intrinsic operations associated with the instantiation of NSs, like booting up virtual machines or setting connections through the transport network interconnecting PoPs. However, the impact of processing federation-related operations and the latency in the link between orchestrators is limited. Ranges of values for each of these deployment schemes can also be taken as reference; for instance, in our setup, federated service creation is on the order of minutes (fewer than five).
There are two main ways in which this article can be extended: first, at the architectural level, by introducing novel distributed ledger technology solutions to automate the exchange of NSD catalogs among peering ADs; and, second, at the algorithmic level, because, once the framework is in place, it can be extended by adding table 2 A description of the processing and operations associated with the NSF procedure.

Component
Description Label in Figure 6 SOEp processing This involves the decomposition of the composite NSD between local/federated nested NSs and iteration over the different nested NSs during the instantiation process.