Verification and validation framework for 5G network services and apps

5G does not only aim at higher capacity and lower latency than current 4G in mobile broadband networks, but also to increase the level of programmability, control and flexibility to meet the requirements from innovative use cases such as IoT, smart manufacturing, and immersive media. However, several difficulties still need to be overcome for a better technological adoption. To reduce the time-to-market for networked services and to lower the entry barrier to third party developers of VNFs and network services, an integrated development and operations (DevOps) methodology is a promising way. One of the biggest challenges in upcoming 5G DevOps is the validation and verification (V&V) of individual VNFs and network services (VNF graphs) so that operators can be sure of their behavior. In this paper, we propose an NFV architecture that supports the DevOps methodology with a V&V platform and an advanced NFV catalogue. The conceptual components of the V&V platform, the foreseen methodology and the outcomes are explained. Finally, we explore some perspectives in applying such V&V approach in another 5G research project.

Abstract-5G does not only aim at higher capacity and lower latency than current 4G in mobile broadband networks, but also to increase the level of programmability, control and flexibility to meet the requirements from innovative use cases such as IoT, smart manufacturing, and immersive media. However, several difficulties still need to be overcome for a better technological adoption. To reduce the time-to-market for networked services and to lower the entry barrier to third party developers of VNFs and network services, an integrated development and operations (DevOps) methodology is a promising way. One of the biggest challenges in upcoming 5G DevOps is the validation and verification (V&V) of individual VNFs and network services (VNF graphs) so that operators can be sure of their behavior. In this paper, we propose an NFV architecture that supports the DevOps methodology with a V&V platform and an advanced NFV catalogue. The conceptual components of the V&V platform, the foreseen methodology and the outcomes are explained. Finally, we explore some perspectives in applying such V&V approach in another 5G research project.

I. INTRODUCTION
5G will not only be an evolution of mobile broadband networks as its predecessors, but also open the door to unique service capabilities [1]. In this context, SDN and NFV are technologies that will shape the evolution of the telecom sector with new network capabilities and business opportunities. The goal is to increase the level of programmability, control and flexibility of networks, while reducing network operations costs. This will bring more service agility, reducing the time to market of services, and more cost-effective service delivery. These features will empower innovation and creation of new emerging applications that 5G promises. This has also caused disruptive shifts for all actors in the classic telecommunications value chain, who will have to adapt their strategy and business models to the new paradigm. Despite the expected benefits of SDN/NFV technologies, several adoption difficulties need to be overcome. Some significant difficulties are the need for interoperability between VNFs and orchestrators (which is being reduced through the open source software community, with efforts such as OSM [2] or SONATA [3]), the combination of SDN and NFV technologies (e.g. lack of flexible support for end-to-end multi-site installations), and the consolidation of the initiatives (to avoid the "additional development needed").
Taking into account the current predictions for the adoption of NFV and SDN, it can be observed that although the technologies are already there, the adoption has only recently started to pick up speed [4]. One of the most promising enterprise NFV use cases is DevOps automation. It is in view of this that we propose an extended NFV architecture that supports a development and operations (DevOps) methodology. DevOps requires a re-tooling of the organisation, as it implies an organisation change in several departments, not only a technological adoption.

A. Need for V&V
One of the biggest challenges in upcoming 5G DevOps environments [3] is the validation and verification (V&V) of virtual network functions (VNFs) and network services (NSs) against different execution platforms so that service operators can be sure that VNFs and NSs behave as expected immediately after they are deployed and put into production. Such a V&V process not only includes functional testing of VNFs and NSs but also non-functional tests, such as performance measurements for gaining insights about resource requirements to fulfil service level agreements (SLAs) and to provide the expected quality of experience (QoE) [5]. To fit seamlessly into the anticipated DevOps workflow, all these V&V procedures need to be fully automated and be able to qualify any VNF or service without further human interaction.
Even though many NFV platforms that support the quick deployment of new network services already exist today [2], [6]- [9], concepts, workflows, and tools to create and automatically test such services are still very limited or completely missing and need to be established to fully support the DevOps methodology for NFV. Some initial approaches for this have been presented by the SONATA [8] project, which provides a set of SDK tools that support service developers to create and ship new network services. This SDK also offers descriptor validation functionalities that go beyond simple syntax or schema checks, e.g. the automatic detection of loops in network function forwarding graphs (VNFFG) [10]. However, all these tests are only done on static descriptors and do not involve the instantiation and execution of the VNFs or services to perform real functional and non-functional tests for qualification. Thus, there are still many manual tests required by developers, system integrators, and service operators to actually check if their services will work correctly on the target platform. Other tools for SDN and NFV testing provide extensive debugging and prototyping functionalities but focus on the initial development phase rather than on automatic qualification of VNFs and services [11], [12].
To overcome this we propose to add a V&V component to the NFV reference architecture which allows all involved parties (cf. Sec. II) to test, validate and verify single network functions or entire network services before they are deployed to production. The proposed V&V system forms a flexible test platform that is able to handle both generic standardised test-cases as well as specialised test-cases that are customtailored to validate, e.g., the compatibility of a service with an operator-specific target platform.

B. 5GTANGO Overview
In the above context, the second phase 5GPPP Innovation Action project 5GTANGO introduces DevOps to SDN/NFV technologies by bringing flexible programmability of 5G networks with: i) an NFV-enabled Service Development Kit (SDK); ii) a Catalogue platform with advanced validation and verification mechanisms for VNF/NS qualification (including 3rd party contributions); and iii) a modular Service Platform (SP) with an innovative orchestrator in order to bridge the gap between business needs and network operational management systems. The project's objectives include: 1) reduce the time-to-market for networked services by shortening the service development cycle and by qualifying those network services to be adopted; 2) enable new business opportunities with the customisation and adaptation of the network to vertical applications' requirements; 3) reduce the entry barrier to 3rd party developers and support the creation and composition of VNFs and application elements as "Network Services"; 4) accelerate the NFV uptake in industry via an "extended" DevOps model and the validation at scale of Network Service capabilities of the 5GTANGO platform in vertical show cases.
To showcase the project results, 5GTANGO is proposing two vertical pilots: smart manufacturing and immersive media.

C. Paper structure
The paper is structured as follows. Section II focuses on the proposed architecture for the V&V Platform and the Catalogue within the 5GTANGO project. Section III focuses on the V&V Platform, including the V&V operation lifecycle, components, and expected outcomes. Finally, section IV concludes the paper and foresees the use of the V&V platform in the FIRE testbed (5GinFIRE).

A. Architecture overview
Our vision is to improve the quality of network services and their availability by introducing verification and validation (V&V) platforms and catalogues. This entire V&V process is performed by the following four roles: • Developer: Develops, tests, publishes network services; • Operator: Selects, tests, deploys existing network services; • V&V pPlatform: Provides testing environment, develops and executes tests, issues digitally signed test results; • Catalogue: Stores network services and metadata (including test results), offers decision support.
There will be both public and private V&V platforms. Public V&V platforms can be hosted by different 3rd parties and offer different test environments. These V&V platforms differ in the tests they offer (e.g., functional vs. non-functional) and the infrastructure they use. It is conceivable that some V&V platforms offer tests for general qualification (e.g., throughput), while others focus and specialize on certain scenarios or infrastructures, e.g., for smart manufacturing. Given a network service and the corresponding payment (if required), a V&V platform executes a set of tests, logs and digitally signs the results and returns the result to whoever triggered the tests.
Developers using 5GTANGO can compose multiple VNFs into a network service. Before publishing the network service, developers use a local V&V platform (with limited functionality) to test their network service. Once satisfied with the local results, developers can perform additional tests using public V&V platforms. Developers can publish the network service together with the test results from such public, trusted V&V platforms in one or more catalogues. More test results can be added later on to increase customers' confidence in the developed network service. Figure 1 shows these roles and their components in more detail.
An operator wanting to deploy a certain network service can search for an existing one in the catalogues. After picking a network service, the operator can perform additional tests using either an in-house V&V platform or public V&V platforms (hosted by 3rd parties). It is up to the operator, whether or not to share the obtained additional test results by uploading them to a catalogue. If the operator is satisfied with the test results, the network service can be deployed in the operators infrastructure. A catalogue holds and maintains submitted VNFs/NSs and corresponding metadata. Based on such metadata, the catalogue can offer decision support that goes beyond simple search and filter functionality. Through ongoing tests or monitoring of already deployed network services and analysis of results, catalogues can continuously update their repository, correlate available information, offering the best and most suitable network services to operators and providing feedback to developers. Obtained benchmark results can also be used by a policy management framework to provide quality guarantees.
Similar to the V&V platforms, there may also be multiple catalogues hosted by different 3rd parties or even operators. These catalogues share the main concept but may differ in scope and provided functionality. They can also have different limitations or regulations concerning the published network services and associated test results to enforce a certain level of quality.

B. The Catalogue
In a typical 5G infrastructure, there is a VNF repository where authorized developers are able to create a VNF and register any metadata to the VNF together in a package. This is the initial form of what we call a Catalogue. The 5GinFIRE project [13] offers such open source repository and via a web portal, users are able to visualize the VNF information and search for VNF. However, this classical repository is unable to provide any information of the confidence level on the VNF behaviours.
Given the need for network services and applications validation, the 5GTANGO catalogue will provide the means for storing and exploiting all information relevant to VNFs/NSs and their validation. The catalogue realizes an enhanced repository both by enabling all store-related operations for VNFs and NSs (i.e. persistent and scalable storage, searching and retrieval of content) and by delivering proposals for optimized operations during deployment and during runtime. The V&V outcomes, the profiling outcomes, information related to policies as well as information related to QoS (through SLAs) will be stored as metadata (i.e., annotations) of the corresponding VNFs. The catalogue will enhance its key functions and interfaces for storing, searching and retrieving VNFs/NSs based on the Fig. 2: General Certification workflow metadata. It should be noted that the metadata will be updated with new V&V results as well as with information from runtime monitoring.
Furthermore, the catalogue is considered to be a multisided catalogue, addressing different stakeholders' needs: It supports developers by providing a repository for storing the developed VNFs/NSs, while also annotating them with additional information (as metadata). The catalogue allows the association of monitoring data with specific VNFs/NSs in order to provide a feedback loop from deployments to developers, offering the possibility of continuous optimization of VNFs/NSs and their deployments.

C. Certification-Oriented V&V
Certifications (or equivalent processes) are organized all over the world in mostly all industrial domains either for regulators or for organizations on a voluntary basis. The aim of a certification process is to ascertain the conformity which is defined as the fact that a product, system, body or even a person meets specified requirements [14] and which can improve the business interests with regard to products, goods and services. The 5GTANGO V&V framework adopts a certification process approach, which means the V&V considers only whether the requirements are satisfied (what should be performed by the VNF/NS), not how they are satisfied. Fig. 2 shows a general certification workflow. The certification authority collects requirements and specifications as subject of certification, and defines the tools and processes to carry out the certification. The certification platform implements the tests to check the comformity with respect to the defined tools and processes. Once an applicant submits his product (i.e. a device, a piece of software, a service), a set of tests are executed on the product in a controlled and trusted environment. The test result is fed back to the certification platform which declares the conformity of the product if the result is satisfactory. A certification can then be issued by the authority.
The proposed V&V framework can be easily mapped to the above workflow. A group of stakeholders, including infrastructure providers and operators, is responsible for defining the specifications and requirements that need to be met by VNFs/NSs. The V&V platform provider develops test cases and test suites, possibly with contribution from developers and other stakeholders, and provides a reference testing environment for test execution. Test results are collected and transfered to the catalogue with basic processing, including transformation to a specific format and attachement of digital signatures. If the results are satisfactory to conclude the conformity of the VNF/NS, the VNF/NS gets a digitally signed metadata set serving as certification.

III. V&V PLATFORM
The main objective of the V&V platform is to perform an adequate set of tests on the VNF/NS targetting the 5G infrastructure and return test results that are digitally signed and include all necessary details. In this section, the lifecycle, the main conceptual components and the expected outcome of the V&V platform will be explained in detail.

A. V&V Lifecycle
This subsection presents the lifecycle of the V&V operations. The main user of the V&V platform in the 5GTANGO concept is the developer. The NS developer creates or reuses available VNFs in order to produce a meaningful NS. The NS comprises VNFs, their lifecycle management (in the form of a VNF manager or modular function-specific plugins) bundled with their own service orchestrator components (or service specific management plugins), packetised along with descriptors for the deployment and configuration of VNFs. The V&V platform that is being developed in 5GTANGO allows the bundling of specific tests along with the service package. These tests are executed from the V&V platform in a sandboxed environment operating under the V&V platform control, so that functional and non-functional metrics are acquired leading to the verification and validation of the NS. These metrics are then stored along with the package in the Catalogue. Additionally, the 5GTANGO SDK will provide the capability (in terms of libraries and helper functions) to the developers to design and describe the tests based on actual infrastructure capabilities. In summary, the lifecycle of service validation and verification is illustrated in Fig. 3.
The 5GTANGO DevOps approach envisages multiple specialised V&V platforms. These platforms may provide complementary or partially overlapping environments for the execution of particular validation and verifications tests. For example, a V&V platform that is linked to a real industrial setup (e.g., a C&C machine) would be in a position to verify services for this particular environment. Based on this particular specialisation (stemming from the availability of certain tools and infrastructure elements), the V&V platforms may also make available standardised tests which can be used to partially or fully validate the NS or its components.  Only blackbox test cases are considered, executed upon the system under test (SUT). Blackbox testing refers to examining the SUT regarding its capabilities without the knowledge of its internal structure. For example, given an input following some specification, blackbox testing verifies whether the SUT behaves correctly and emits the expected output. In 5GTANGO, specifications and requirements come from standard specifications (e.g., ETSI-NFV), the use cases scenarios, and deployment environments. The perimeter of the SUT needs to be clearly determined (e.g., whether the composed NS is to be tested or just a single VNF) in order to apply appropriate blackbox tests. Functional and non-functional tests are both possible using the blackbox approach. • Test categorisation and test suites. Generated blackbox test cases are stored in a repository with needed tags to facilitate the test discovery and search. These tests are configurable such as the SUT access information, the target value of throughput. The tags contain also information about the test categorisation, such as "functional test", "performance test", "packaging test". A test suite is a specific set of test cases chosen from the test repository in order to check whether the SUT has met all the requirements of the target infrastructure, or of some pre-defined use case, e.g. smart manufacturing. Additional test cases can be included in the test suite if the infrastructure provider decides to do so. • Reference testing environment. To perform the blackbox tests mentioned before, an execution environment is required in which the SUT is deployed and executed and can be stimulated, for example, by sending test traffic to its interfaces and observing the outputs. We explicitly do not fix our V&V platform to a single test execution environment but allow it to interface with multiple different environments ranging from small emulated platforms [12] to full-featured network function virtualization infrastructure (NFVI) setups [15] which may be configured as generic testing platforms following pre-defined standards or operator-specific guidelines. For example, a small emulated execution environment running on the service developer's laptop only offers support to do very simplistic functional tests, e.g., to check if configuration interfaces of a NS work like expected. Those platforms can in particular not be used to do non-functional performance tests. For those tests, either platforms which follow a pre-defined hardware setup are required to allow the comparison of performance results, or an operator needs to connect a V&V instance with his own environment to do performance tests directly on the intended target platform. In each case, information about the environment in which a particular set of tests was executed has to be stored together with the test results to improve replicability. • Test result repository. After the test suite has been executed upon the SUT within the reference testing environment or operator-specific one, the test result is stored in a repository with all the details, such as the test case reference, the test environment parameters, the resource allocation information. The result will input to the catalogue to be compiled into VNF/NS metadata. • Monitoring platform for V&V. In order to cope with the requirements of the DevOps paradigm, the V&V platform needs to support efficient instrumentation for allowing the verification and validation of services both in vitro and in situ. In order to achieve that the monitoring framework that will support the V&V platform will support a hybrid monitoring scheme utilising legacy passive monitoring operations with active ones, based on deployment of probes (as VNFs) in specific points within the desired service chain. The placement, configuration and operation of such probes will be defined and controlled by the developer. The monitoring platform will operate in (at least) two environments, one being the sandbox environment where the developer is deploying the service components for testing and the production/demonstration environment where the service is deployed for operation. These enhancements facilitate cross-layer mechanisms for supporting functionalities of the extended 5GTANGO DevOps approach, addressing different levels: (i) VNFM: 5GTANGO will exploit the flexibility offered by the autonomicity concept in order to offer reusable software defined decision elements (DE) that can be exploited by developers and modified at runtime supporting the DevOps model. (ii) VNF: 5GTANGO will support through its SDK the embedding of layer 2-3 and knowledge plane DEs in the VNFs in order to allow the collection of VNF specific metrics supporting the autonomic management of

C. V&V test development and execution framework
In the proposed V&V platform, we decide to use TTCN-3 for test development, generation and execution.
Testing and Test Control Notation version 3 (TTCN-3) is a test scripting language widely known in the telecomunication sector. It is used by the third Generation Partnership Project (3GPP) for interoperability and certification testing, including the prestigious test suite for Long Term Evolution (LTE)/4G terminals. Also, the European Telecommunication Standards Institute (ETSI), the languages maintainer, is using it in all of its projects and standards initiatives, like oneM2M. A test case produced by TTCN-3 is abstract, because it does not know how or where to send the content, it only knows that it has to send it. This is where the TTCN-3 Control Interface (TCI) and the TTCN-3 Runtime Interface (TRI) get into the process. The TCI is composed of Test Management (TM) module, Test Logging (TL) module, Coding and Decoding (CD/Codec) module and Component Handling (CH) module. The TRI contains System Adapter (SA) and Platform Adapter (PA) modules. All of these modules are already provided by most TTCN-3 tools, except the Codec and the SA modules. The first is needed to convert TTCN-3 structures to a binary, text, XML, JSON or some other serializable format, and the latter is needed in order to communicate with the SUT, because it implements the protocols how the converted data from the codec can be sent or received from the SUT (for example convert the structure to a HTTP request or response, if the protocol used is HTTP).
TTCN-3 test cases can be generated manually or automatically using advanced methods such as model based testing (MBT) [16], the latter improving the traceability of tests in respect to requirements and easing the maintenance of the repository, at the expenses of strongest skills needed to develop the test model.

D. Outcome and Benefit from V&V
• A "star" system. A star-level is an abstraction of test result of the VNF/NS just tested. It aims to give an overall confidence level on the VNF/NS rather than to substitute the detailed test result towards the Catalogue. The confidence level comes from two parties: the first one is the target deployment infrastructure and the second one is the user who discovers them in the Catalog and uses them for his own experiment/application. The former gives a star level based on the test result whereas the latter rates the VNF by a star-level to express his satisfaction. These stars can be used later sorting the VNFs as most populars using rating and download metrics. • An information-driven catalogue. The proposed catalogue will not only store VNFs/NS but also annotate them with rich representations (i.e. metadata) that will be utilized for all its operations. These metadata refer to information obtained from the SDK (e.g. version, developer and licensing information, etc), from the V&V components (including test cases and test outcomes), from the platform (such as deployment configurations), from the infrastructure components (e.g. monitoring data, QoE data, etc.) and from the policy management framework (including policies and SLAs). The catalogue is considered to be an information-driven catalogue given that all operations will be performed based on the corresponding metadata, as for example searching for VNFs or correlating information for different VNFs of an NS. • Two added-value services providing decision support and continuous optimization. The proposed decision support service aims at supporting the infrastructure owner/operator regarding the selection and use of assets (i.e. VNFs and NSs). Given that several information sources will generate data, the goal of this service is to enable their understanding and optimum exploitation by the operators. To this end, it will analyse all stored metadata (as described in the previous paragraph) and propose specific VNFs/NSs based on the operators' requirements (e.g. QoS/QoE parameters). The continuous optimization service will introduce dynamicity in the overall V&V process by allowing not only static (i.e. pre-defined) test configurations to be executed, but also by considering the actual outcomes and behavior of the deployed VNFs/NSs (through the actual deployment information and the collected monitoring data) and proposing new V&V tests to be executed accordingly.

IV. CONCLUSION AND PERSPECTIVES
The proposed V&V framework and the catalogue realise a complete environment for validating and verifying VNFs/NS and for exploiting in an optimum way the outcomes of this validation and verification process by correlating them with additional data emerging from other architectural components. The added-value services of the catalogue will introduce dynamicity and automation in the overall process by correlating and analysing the aforementioned data and proposing the corresponding actions. This V&V framework and catalogue are designed in a generic way in order to be adapted easily to other 5G infrastructures, such as those in 5GinFIRE. For 5GinFIRE, the validation and verification of VNFs targeting the 5G testbeds is out of the scope of the project, however, it is important for 5GinFIRE platform users to know beforehand if the VNFs are certified for features and performance. 5GinFIRE is eager to apply and adapt the 5GTANGO V&V platform and catalogue to complement its work.