Multi-Domain Orchestration of 5G Vertical Services and Network Slices

This paper presents an orchestration framework for the delivery of 5G vertical services and end-to-end network slices in a multi-domain scenario. The proposed architecture relies on a business model where verticals and service providers take the roles of digital service consumers, digital service providers and network service providers, each acting in its administrative domain. The interactions and delivery procedures among these entities leverage on standard solutions for interfaces and information models defined by ETSI and 3GPP. The paper also presents proof-of-concept applications of the proposed architecture in two H2020 European research initiatives.


INTRODUCTION
Network Slicing has definitely emerged as the key enabler in the paradigm shift from 4G to fully Network Function Virtualization (NFV)/ Software-Defined Networking (SDN)enabled 5G era. 5G mobile systems will in turn enable network operators and service providers to host vertical industry segments by introducing new services and enhance business collaborations between providers and customers at large. Moreover, vertical end-to-end (E2E) services can benefit from a large coverage area across different operators and providers, as well as technical and administrative domains. Therefore, slice and service deployments need to be able to span across different domains, requiring new automated mechanisms for provisioning E2E slices targeting a tight coordination of multi-domain coordination functions.
In this context, it becomes clear that orchestration is one of the most important aspects in today's service and network providers operations. It addresses many important aspects, such as exposing service offers, provisioning new services and slices and coordinating heterogeneous resources (network, radio, cloud) according to the service constraints. Indeed, orchestration procedures are paramount to achieve the desired service automation and vertical driven requirements. Moreover, the orchestration procedures have to consider vertical E2E services composed by Network Slices (NSs) delivered in multiple service provider domains.
On the other hand, emerging trend of closed-loops automation, enhanced by Artificial Intelligence (AI) models, is to be considered as the enabler to proactively maintain the desired service levels to the verticals. In this regard, closing the management and/or control loop requires orchestration to be agile and flexible enough to deal with real-time scenarios brought by the cognitive network management.
Finally, 5G networks require orchestration procedures to integrate several heterogenous paradigms and resource types, including NFV, SDN, Multi-access/Mobile Edge Computing (MEC), among others, when interacting with the network environment.

II. NETWORK SLICE ORCHESTRATION: RELATED WORK
The work related with 5G vertical services and NS orchestration is being addressed by several Standards Development Organizations (SDOs) and Open Source Communities (OSCs), while the telco industry is approaching with commercial solutions. Starting with SDOs, the 3GPP provides the definition of Network Resource Models (NRMs) [1], which lays the foundations for the management and orchestration of 5G networks, including New Radio (NR), Next Generation Radio Access Network (NG-RAN), 5G core network (5GC), and NSs as well. Complimentary to this, ETSI has proposed several architecture standards that cover important aspects about network and service orchestration in 5G-enabled infrastructures. Specifically, ETSI NFV defines an NFV architecture framework that is focused on the lifecycle and orchestration of Virtual Network Functions (VNFs) and network services built by composing them, which are considered as NS components. Additionally, ETSI Multiaccess Edge Computing (MEC) specifications are defined to enable a self-contained mobile edge cloud able to exist in different cloud environments. In most telco environment the real requirement is to extend NFV approach towards the mobile edge concept, due to the maturity of the NFV specifications and initial compliant products and multi-vendor interoperability trials. For this reason, ETSI MEC has defined a MEC-in-NFV reference architecture (GS MEC 017 [2]) where the NFV management functions provide orchestration of services for mobile edge scenarios. In the open source community, ETSI OSM (Open Source MANO) [3], which is a community-driven project led by operators, is developing an Open Source NFV Management and Orchestration system aligned with ETSI NFV. The main goal of OSM is to achieve E2E network service deployment for telco services through the orchestration and life-cycle management of VNFs and local network services. ONAP (Open Network Automation Platform) [4] is an open source software platform that delivers capabilities for the design, creation, orchestration, monitoring, and life cycle management of: i) VNFs; ii) carrier-scale Software Defined Networks that contain them; and iii) higher-level services that combine the above. It provides automatic, policy-driven interaction of these functions and services in a dynamic, realtime cloud environment. Finally, the telco industry is eager to adopt orchestration solutions for the deployment of heterogeneous vertical services and NSs across several technology domains. Among others, Ericsson [5] and Nokia [6] are key players in the NS orchestration area, and both are offering E2E slice orchestration solutions to their customers to facilitate the delivery of customized 5G services.
The 5G vertical services and network slices multi-domain orchestration approach proposed in this paper leverages on the mentioned existing work developed in the standards and open source communities.

III. MULTI-DOMAIN ORCHESTRATION OVERVIEW
The network/service lifecycle management paradigm involves three main activities: 1) Assurance (monitor and collect raw metrics; aggregate metrics to produce rich indicators; correlate information to produce richer events); 2) Analysis and Decision (analyze network/service indicators to detect problems and then decide eventually on corrective actions); 3) Fulfillment (orchestrated action enforcement).
To deliver E2E communication capabilities to verticals over multiple administrative domains, multiple business roles have to be considered, as depicted in Fig. 1. The Digital Service Customer (DSC)/Vertical subscribes and consumes communication services from the Digital Service Provider (DSP); the Vertical should indicate only the required information about the service (e.g. bandwidth, latency requirements, etc.), without having to understand the network details and how the service will be composed and offered. The DSP is responsible for managing the service lifecycle, including their creation and exposition to the Verticals, as well as their decomposition (during provision) into one or more NSs across one or multiple administrative domains. The Network Service Provider (NSP) is responsible for managing the NSs and involved network resources lifecycle, including their creation and exposition to the DSP, as well as their provision, monitoring and optimization.
On the other hand, orchestrating E2E NSs composed by several NSs offered by multiple administrative domains poses several challenges. First, the business entity (DSP) providing the E2E NS towards the Vertical needs to have detailed information about the NSs offered by each one of the NSPs. Therefore, it is paramount that well-defined cross-provider (DSP with NSPs) mechanisms are available for NSPs to offer their NSs towards the DSPs. With this information in place, DSPs orchestration platform is able to consume NSP NSs offers and design complex E2E NSs according to their business aims. Another critical challenge is to deal with the E2E NSs provisioning process. In this case, according to the E2E NS subscribed by the Vertical, as well as depending on specific customer requirements (e.g. E2E NS mobile or fixed network endpoints), DSPs will have to select the most appropriated NSP NSs and request its instantiation towards the corresponding NSPs. This requires well-known crossprovider provisioning Application Programming Interfaces (APIs) to enforce the DSP-NSP NS provisioning requests. Thereafter, during the E2E NS runtime, the DSP must have the capability to periodically be informed by each NSP about the NS performance. Again, this requires well-established monitoring APIs between the NSPs and the DSPs, allowing the later to collect NSs counters and/or Key Performance Indicators (KPIs), and produce E2E NS performance dashboards.
The E2E NS performance information is very relevant to address another critical aspect -guarantee that the contracted Service Level Agreements (SLAs) with the Verticals are being delivered as promised. Maintaining the E2E NS performance is also a responsibility of the DSP. Towards this objective, the DSP orchestration platform should have in place E2E optimization mechanisms to guarantee the sustainability of the E2E NS. This requires advanced analytic procedures, such as AI/Machine Learning (ML) mechanisms, to anticipate potential faults in the contracted NS and proactively actuate (e.g. replace a potentially faulty NS by a similar NS delivered by another NSP) to guarantee that the E2E NS is not impacted.

ARCHITECTURE
The proposed multi-domain orchestration is based on a cross-layer combination of three levels of orchestration logics, namely for vertical services, NSs and slice resources. These levels of orchestration logics are implemented in three different functional modules which are mapped to the business roles identified in section III: the Service Orchestrator at the DSP level, the Slice Orchestrator at the NSP level and the Resource Orchestrator at the NSP level. While the combination of the Service and Slice Orchestrators builds the SS-O (Service and Slice Orchestrator), the Resource Orchestrator is defined and described hereafter as NMR-O.  Fig. 1 shows how these orchestration function modules relates each other to build the proposed multi-domain orchestration architecture for 5G vertical services and NSs. In practice, each NSP domain offers a set of NSs to one or more DSP domains. The Slice Orchestrator within each NSP domain is responsible to expose single domain NSs, managing their lifecycle (instantiation, configuration, runtime operations) leveraging on the capabilities offered by the NMR-O for the NSP domain resource orchestration (at network, NFV and MEC level). In turn, each DSP offers to verticals E2E NSs, spanning across multiple NSP domains, in the form of 5G vertical services. The following subsections provide more details and functional insights on the three main building blocks of the proposed architecture. First, a brief overview of the multi-domain orchestration information model and interfaces that regulate the interactions among the three main architecture building blocks is provided in section IV.A.

A. High-Level Information Model and interfaces
To enable the proposed multi-domain orchestration solution, the information model regulating the interactions among the three main architecture building blocks follows a layered approach, where vertical services, NSs and slice resources are well defined, where possible, adopting standard models available from 3GPP and ETSI. Therefore, the multidomain orchestration information model can be split into three main interworking and integrated components: vertical services, NSs, slice resource.
The vertical service information model, that regulates the Service Orchestrator coordination logic, is built around the concept of Vertical Service Blueprint (VSB) and Vertical Service Descriptor (VSD) [7]. The VSB is a high-level description of a template for an E2E NS service. It includes a high-level description of the atomic functional components (NFV Network Services, VNFs, Physical Network Functions (PNFs)) of the service and their interconnection. Moreover, it contains a set of service parameters (e.g. to correctly size the service in terms of resources and requirements) that have to be filled by the vertical at instantiation time to customize the E2E NS. The VSDs are the vertical service descriptors that drive the provisioning of new vertical service instances (VSIs), and are obtained after parameterizing the VSBs in the parts to be filled by the vertical for service customization. A VSI is therefore logically representing an E2E NS instance. On the other hand, the NS information model follows the 3GPP NRM [1]. In particular, the proposed slice information model, that is at the base of the NSP NS lifecycle management (in the Slice Orchestrator), leverages on the NS part of the 3GPP NRM, with focus on the Network Slice Instance (NSI) and Network Slice Subnet Instance (NSSI). Following this 3GPP standard approach, the NS information model is built around four main components.
The Network Slice Template (NST) is a descriptor of the overall single domain NS offer of an NSP, and therefore includes a set of attributes that characterize the network slice. Specific attributes for the slice access points are included to describe how the NSs created from the NST can be accessed and interconnected to other slices. The Network Slice Subnet Template (NSST) details the capabilities and requirements of the individual components of the NSs offered by a given NSP. Following the 3GPP NRM approach, NS subnets can be implemented as NFV Network Services. The Network Slice Instance (NSI) models a provisioned NS in a given NSP domain, instantiated based on an existing NST. The NSI information model is based on the 3GPP NRM NS definition [1]. The Network Slice Subnet Instance (NSSI) represents a constituent logical component of an NSI and its model is also based on the 3GPP NRM NS subnet definition, and includes slice subnets attributes and characteristics in terms of Managed Functions (which can be VNFs and PNFs) and NFV Network Services. The slice resource information model provides the means to define the Network Services that compose an NS or an NSS. To do this, the slice resource information model relies on the Network Service Descriptor (NSD) and the Virtual Network Function Descriptor (VNFD) entities defined by ETSI [8] [9]. The NSD contains the information used by the NFVO for the lifecycle management of a Network Service, which is, in turn, a composition of VNFs. These VNFs are modelled by means of the VNFD.
Based on these information models, the Service Orchestrator, the Slice Orchestrator and the Slice Resource Orchestrator interact for the provisioning of 5G vertical services and NSs through a well-defined set of interfaces. The vertical service management interfaces are exposed by the Service Orchestrator as Representational State Transfer (REST) APIs and implement the following vertical service lifecycle management operations: 1) Query VSBs; 2) Create, query, update, and delete VSDs; 3) Instantiate, query, modify and terminate VSIs; 4) Actuation for E2E NSI optimization; 5) Notifications about vertical service lifecycle events.
On the other hand, the NS management interfaces are exposed by the Slice Orchestrator as REST APIs (aligned with those defined by 3GPP in [10]) and support the following lifecycle management operations: 1) Query NSTs and NSSTs; 2) Instantiate, query, modify and terminate NSIs and NSSIs; 3) Notifications about NSIs and NSSIs lifecycle events; 4) Subscriptions and notifications for NST related events. Similarly, the slice resource level interfaces are aligned with ETSI [8] and provide support to the lifecycle management of the Network Service Instances and their associated VNFs, which includes instantiation, querying, modification, termination and notification of events.

B. SS-O architecture
As described in the previous sections, the SS-O functionalities are implemented by the combination (and interaction) of the Service and Slice Orchestrators, which respectively reside in the DSP and NSP domains. The Service Orchestrator at the DSP level provides orchestration of vertical services as requested by verticals and takes care to map these into E2E NSs which are composed by singledomain NSs offered by one or more NSPs. Therefore, it manages the lifecycle of E2E NSs and coordinates the Slice Orchestrators. On the other hand, the Slice Orchestrator is responsible for the coordination of single-domain NSs lifecycle, thus interacting with the NMR-O resource orchestrator for the actual creation and configuration of slice resources. The Slice Orchestrator is deployed at the NSP level, and it manages single administrative domain NSs while, in terms of management logics, it operates at a finer resource granularity with respect to the Service Orchestrators within the DSPs, since it coordinates the actual resources in the NSP physical and virtual infrastructure.

1) Service Orchestrator functional decomposition
The high-level functional split of the Service Orchestrator is shown in Fig. 2, where the main service coordination functions are identified along with their interactions. In terms of vertical service and E2E NS management features offered by the Service Orchestrator, they can be summarized as follows: i) interaction with verticals for service offer exposure and customization, ii) collection of NS offers from NSPs, in the form of NSTs, iii) lifecycle management of vertical services, mapped to E2E NSs and composed by single domain NSs, iv) maintenance of up to date vertical service and E2E NSIs status and features, v) coordination of multi-domain actuation operations for runtime optimization of E2E NSs. In the following, a brief description of each functional module of the Service Orchestrator is provided.

2) Slice Orchestrator functional decomposition
The high-level functional split of the Slice Orchestrator is shown in Fig. 3, where the main slice coordination functions are highlighted with their interactions. As part of its singledomain NS management responsibilities, the Slice Orchestrator provides the following main functionalities: i) interaction with DSPs for single domain NS offer exposure, including slice components and capabilities, ii) onboarding and maintenance of NSTs and NSSTs, iii) lifecycle management of NSIs and NSSIs, including mapping and translation of NS requirements into resource constraints and operations, iv) maintenance of up-to-date NSIs and NSSIs status and characteristics, v) coordination of single-domain actuation operations for runtime optimization of NSs. In the following, a brief functional description for each of the components depicted in Fig. 3 is provided. NST/NSD Translator: it translates (at instantiation time) the NS level requirements expressed in the NSTs into finer grain resource level requirements. In practice, it associates an NST with a set of specific slice configurations and NFV NSDs. Arbitrator: it is the decision maker for regulating the slice resource sharing among different NSs. It evaluates if some of the existing (i.e. already provisioned and configured) slice resources can be reused without affecting the running services and slices, by matching the NST requirements and constraints against the available NSIs. Network Slice Inventory: it stores and maintains up to date information related to the provisioned and running NSIs and NSSIs, including relevant characteristics and configurations of slice resources, like NFV Network Service Instances and reference to the instantiated Network Functions (NFs). Southbound resource control drivers: interactions with technology-specific resource orchestrators are required for slice resource and configuration according to NSTs constraints. For this, southbound drivers are included to implement specific APIs exposed by the slice resource orchestrators, like the NMR-O.

C. NMR-O architecture
The NMR-O stands for NFV, MEC, RAN Orchestrator and it has been designed to act as the network domain orchestration, that is, the entity responsible for managing the network services lifecycle and the resources associated with them. The NMR-O implements the following resource coordination functionalities: i) exposure of NFV Network Service offers to the Slice Orchestrator, ii) onboarding and maintenance of NSDs and VNFDs, iii) lifecycle management of the NFV Network Services composing the NSs, iv) up-todate maintenance of the status and characteristics of NFV Network Service instances (and VNFs), v) Support NFV Network Service instance monitoring.  Fig. 4 depicts its functional architecture. The main goal of the NMR-O is to configure and maintain the atomic elements that compose the NS, which is basically a set of virtual and physical functions that are configured and interconnected at the infrastructure of the NSPs. The NMR-O receives NS requests from the SS-O at NSP level through the Northbound Interface (NBI). These requests are firstly processed by the Network Service Orchestrator (NSO), which splits them into the lower level configurations required by the NFVO (which is implemented by ETSI OSM [3]) and the Extended Infrastructure Manager (EIM). OSM is based on the ETSI NFV Architecture proposed in [11] and focuses on covering the VNF Manager component, which is related to the VNF lifecycle management, and the NFVO, which handles the general resource orchestration and network service lifecycle.
Here, OSM provides the orchestration of the VNFs composing the network services requested from the SS-O. In this regard, the NSO forwards the network service related operations received through its NBI to the OSM NBI. Such requests are based on the NSDs and VNFDs supported by OSM, which are compliant with the ETSI standards. Upon the reception of these requests, OSM contacts the Virtual Infrastructure Manager(s) (VIMs) of the NSP to allocate and configure the computational and network resources to provide the network service(s). The EIM aims at implementing the operations needed to configure the full network service in support of the slices provided by the SS-O. In particular, additional configurations may be needed to interconnect independent network service instances associated to the same slice, or to provide network service instances connectivity to services allocated in a different domain (shall it be different VIM, operator, NSP, etc.). To do this, the EIM contacts the infrastructure manager of the NSP.

A. The SliceNet cognitive management approach
The H2020 SliceNet project [12] has designed a cognitionbased approach to provide the tools for achieving Quality of Experience (QoE)-aware management of multi-domain NSs built on top on the proposed orchestration approach. This has been achieved through the implementation of a full Monitoring, Analysis, Planning and Execution governed by a Knowledge-base (MAPE-K) loop (cognition system) that encloses QoE monitoring and analysis functions as well as an actuation system to apply remedies based on the monitoring/analysis outputs for the NS QoE maintenance. To achieve this purpose, the orchestration system interacts with the cognition system as the entry point to enforce the desired actuations to guarantee the QoE of the multiple deployed E2E NSs. SliceNet has considered multiple use cases (UCs). Here we focus on the e-health UC, that aims to provide a NS for the vertical (hospital) offering the best services and an excellent QoE to a paramedic team. A smart connected ambulance is roaming through the NSPs while sending data streams that should be delivered in real time with no quality degradation. A feedback mechanism is implemented, allowing for verticals to express their perceived quality of the service at will. In addition, the cognitive system relies on an anomaly detection ML model to predict if a degradation in the network signal strength in the RAN segment may be perceived in the future 5 minutes by observing the last 5 minutes of Quality of Service (QoS) metrics retrieved from the User Equipment (UE) connected to the deployed NS supporting the eHealth service. Both mechanisms allow to correlate the service quality with the actual quality at the deployed network resources, either slices or VNFs (see Fig. 5). In case a bad quality situation is detected, the cognition system engages with the orchestration system presented in section IV to enforce modifications to the deployed NS as to maintain optimal QoE levels. In this case, the Service Orchestrator acts as the entry point for all re-optimization operations coming from the cognition system. The Service Orchestrator identifies which is the affected E2E NS and determines which of the single domain NSs need to be modified to enforce the optimization operation. Next, the involved Slice Orchestrators are contacted, specifying the operation that is needed as well as its parameters (e.g. new bandwidth to be provisioned). The Slice Orchestrators coordinate the underlying operations so as to materialize the single domain modifications necessary at NSP level to apply the decided E2E remedial action. This may imply changes on the deployed NSs or modifications to the virtual resources, for which interactions with the NMR-O would be required, this being the responsible for the orchestration of the virtual infrastructure.

B. The 5Growth approach
The H2020 5Growth project [13] builds a multi-domain environment for the technical and business validation of 5G vertical services in the area of Industry 4.0, Transportation and Energy. The project follows a field-trial-based approach, where the target infrastructure integrates vertical-owned sites and E2E 5G experimental and validation platforms. The 5Growth architecture leverages on the solutions for vertical service management, network slicing and multi-domain federation described in [14]. However, the concepts of multidomain federation, originally handled at the Service Orchestration level, have been extended to support additional models where E2E vertical services are composed and delivered across multi-site infrastructures through the cooperation of multiple service providers and/or NS providers. In particular, the multi-domain management is implemented also at the Vertical Slicer component, the functional entity responsible for the provisioning of vertical services through a suitable and efficient set of NSs. This approach follows the multi-domain solution proposed in this paper and allows to deal with the specific requirements of the interaction between domains administered by the verticals, where the local system has full control of resource management, virtualization, service and slice provisioning, and external 5G experimental and validation platforms that provide high-level interfaces to request vertical services or NSs.
As shown in Fig. 6, 5Growth proposes two options to support multi-domain at the Vertical Slicer level. The former is based on the multi-domain communication service concept. The Communication Service Management Function (CSMF) entity is extended with a Communication Service Federation Function (CSFF) that decomposes the E2E VSB into a number of vertical "sub-services", where each of them can be provided by the local domain or through federated sites. This case is characterized by a peer-to-peer interaction between Vertical Slicers, with interfaces based on the advertisement of VSBs offered by each site and requests for vertical "sub services". The second option is based on the multi-domain NS concept, where a single vertical service is mapped into an E2E NS composed of multiple NS subnets, as specified in the NST. The NS subnets can be instantiated in the local domain through the NS Management Function (NSMF) of the entry point Vertical Slicer or they can be provided through external sites. In this case, the interaction between the Vertical Slicers deployed in each domain is based on a hierarchical approach. The CSMF of the Vertical Slicer handling the E2E service request interacts with the NSMF of the Vertical Slicers deployed in the other domains. The interface is based on the REST API offered by the NSMF, which includes mechanisms to advertise the NSTs available in the NSMF catalogue and to handle the lifecycle management of NSs.

CONCLUSIONS
This paper has presented a multi-domain standard-based orchestration platform able to cope with the challenging requirements posed by vertical-oriented 5G services and NSs. Two proof-of-concept cases, deployed in the context of two H2020 European research initiatives, illustrate the application of such architecture. In SliceNet, the orchestration plane enables a vertical-oriented QoE-driven network slicing framework that focuses on cognitive network management. In 5Growth, it coordinates the automated technical and business validation of 5G technologies and vertical services in a fieldtrial-based approach.