Test Recommendation for Service Validation in 5G Networks

. System Testing is the testing of a fully integrated software, as well as a series of different tests whose purpose is to exercise the full computer-based system. Software testing though, is an arduous and expensive activity. In the context of manual testing, any effort to reduce the test execution time and to increase defect findings is welcome. One of the biggest challenges in the upcoming 5G era is the validation and veriﬁcation of VNFs and NSs, so that operators can be sure of their behavior in different execution environments. 5G technologies as well as SDNs are already introducing great challenges in the field service validation, testing and optimization of the underlying network. Considering the above, we propose a recommendation system acting as a Decision Support Mechanism integrated in an SDN environment. Its purpose is the proposition of tests to the software developers towards a stable NS ready to be deployed to the production environment.


Introduction
Undisputedly, the rapid evolution of mobile communications delivers a remarkable impact on the social and economic development. The design of the fifth generation (5G) mobile wireless standard introduces the reconstruction of the prevalent network infrastructure and the enhancement of the wireless connectivity [1]. It is of no doubt that the advent of 5G will provide the support of a plethora of innovating services with improvements in system capacity and spectrum coverage. With the aim of bridging the gap between the capacity improvement and scalable connectivity of the multiple users, the advent of 5G mobile networks will introduce paramount changes from the perspective of flexible network management [2]. The Telecommunications Service Providers (TSPs) anticipate demanding requirements from the 5G network, such as high throughput, low latency, increased capacity, and unlimited scalability. The principles of Software Defined Networking (SDN) and Network Function Virtualization (NFV) can effectively diminish these challenges with aligned optimization of short-service life cycles, playing a pivotal role with the respect to the commercialization of 5G networks [3]. In the 5G ecosystem, the exploitation of SDN and NFV concepts offers agility enhancement and maintenance automation compared to the legacy technologies, while it provides the necessary tools to simplify the vertical system organization [4,5]. Through the virtualization of lower level functions, SDN and NFV supplies the necessary tools for the composition of the 5G architecture with network functionalities in software components, namely the Virtual Network Functions (VNFs), through their deployment onto standard devices [6]. The combination of VNFs as Network Services (NSs) can be twinned with respect to the target performance parameters and mapped onto the infrastructure resources. The optimum design and management of the VNFs leverages the peak benefits of NFV landscape of reducing the Capital Expenditure (CAPEX) and Operating Expense (OPEX) in 5G networks, by means of dynamic resource allocation and traffic load balancing [7,8].
As 5G network services and applications are key in several enterprise domains, VNF/NS developers and consumers are still lacking tools to verify functional and nonfunctional properties of these services. Such tools should consider properties such as functional correctness, service availability, reliability, performance and security guarantees. The close interaction between high-integrity VNFs and VNFFGs, with their operating environments (SDN/NFV/MEC Infrastructures), places a high priority on understanding and satisfying both functional and nonfunctional requirements of their configuration [37].
However, in this plethora of the NSs and VNFs, a big challenge of choosing the appropriate test for validating and/or verifying the on boarded NSs arises. Due to the many test combinations it seems impossible to test each and every combination both in automatic and manual testing. Also, many times, test developers need to communicate with the service developers in order to discuss the service requirements. If the test developer fails to completely understand the service's needs, then it's mostly sure that the chosen test won't be the correct one. Furthermore, most of the testing projects are trying to translate their manual test cases to automated ones, in order to improve productivity and coverage. One of the key steps to commence automation testing isselecting the appropriate test cases and determining the Return on Investment (ROI). Automation though, does not overpower or replaces manual testing, but it compliments it. Like manual, automation to, needs a strategy with proper planning, monitoring and control. Automation, when implemented correctly, can become an asset to the team, project and ultimately to the organization.
In this paper, moving towards the automated testing, we present a Decision Support Mechanism (DSM), which is based on a recommendation system, aiming to support the test developers at the selection of the most relevant test cases, for a given NS. The presented mechanism is integrated in a real 5G environment, the SONATA (powered by 5GTANGO) development and validation platform. The DSM, as part of the Validation and Verification Platform (V&V) of 5GTANGO, aims at providing a set of recommendation tests to the end users, when they choose to trigger a test for a specific NS.
The rest of this paper is organized as follows. Section 2 presents the state of the art regarding software defined networks, network service validation and recommendation systems in general. Section 3 describes the general architecture of SONATA (powered by 5GTANGO), but also how the recommendation engine is positioned into the V&V platform. Section 4 presents the internal architecture of the proposed DSM and the algorithms used for the provided recommendations.

2
State of the Art

Software Defined Networks
Software Defined and 5G Networks -a combination of SDN and NFV-is a significant area that tend to make NSs and applications more flexible. NFV decouples network functions from the underlying hardware and centralizing them at network servers making the network architecture highly flexible from quick and adaptive reconfiguration [9]. The services' complexity and specialized requirements could be easily handled by transforming them to NFV and deploying the underlying applications in a software-defined environment. In addition, it is worth mentioning the fact that many service providers are anticipated to adjust network operations to promote a business model where NSs and network resources are managed and verified properly. SDN brings ease of programmability in changing the characteristics of whole networks. This simplifies the management of the network, since it is decoupled from the data plane. Therefore, network operators can easily and quickly manage, configure and optimize network resources with dynamic, automated and proprietary-free programs [10]. VNFs are at the heart of NFV that manage specific network functions such as firewalls and load balancing. In order to establish a completely virtualized environment, individual VNFs can be chained as building blocks, namely NSs [11]. VNF transfers network functions from dedicated hardware devices into a software layer running on commercial off-the-shelf servers (COTs), offering a huge reduction in network costs. Improved mobility and faster time-to-market for new services is also an advantage [12]. Moving a step forward, there are also many NFV platforms that support the quick deployment of new NSs while they also try to validate and verify them [13,14,15].

Network Service Validation
One of biggest challenges facing the 5G environments is the validation and verification of VNFs and NSs against various execution platforms so that network providers can be confident that services perform as anticipated. Such V&V processes not only include functional testing of VNFs and NSs but also non-functional tests, for example performance measurements [16]. One could define validation as the ability to predict the impact of different actions, reason about correctness, and diagnose unexpected network behavior, for a selected VNF or NS [17]. There are many different tools for the verification of NSs. Batfish [18] is an open source network configuration analysis tool which is used for verification in pre-deployment taking advantage of model-based analysis. Moreover, GNS3 [19] and vrnetlab [20] are tools for non-functional testing also in pre-deployment, with which test developers can deploy configurations and check the resulting network behavior through emulation.

Recommendation Systems
A Recommender System (RS) is a software application or algorithm, which collects and analyzes the preferences (i.e. ratings) given by users aiming to suggest relevant items to the interested users (e.g. books to read, mobile applications to download, VNFs to instantiate or even tests to validate a whole NS) [21]. RSs are moving a step forward since recommendations can be of various types of items, depending on the underlying domain (such as web sites, applications, software, VNFs, NSs or test functions). A recommendation system plays an extremely important role in the emerging SDNs and especially in NS validation and verification platforms, since it helps the platform to promote validation flows to the test developers and finally lead to greater commercial benefits. Currently, collaborative filtering is the most widely used commercial recommendation algorithm.

Related Work
In parallel with the vast development of the web, the arrival of the rapid expansion of the information available to the users became apparent. Recommender Systems pioneered the web with the goal of integrating contextual data and offering practical feedback to the end-user. While the research field of RSs has expanded in various fields, there is a steady interest in the application of RSs to SDN environments. In this context, a predictive, automated network management recommendation was also introduced with surprising results in [22]. The main driver of these recommendations was the appropriate trading policies to enforce the recommendation module and to propel the actions required to get the best value per VNF. However, the absence of RSs in SDNs is crucial in the tacit use of the multiple quality of service-oriented elements, such as monitoring systems, regulation, VNF/NS storage. RSs approaches and procedures therefore include a range of tools that must be thoroughly examined. An approach is discussed in [23], where authors proposed a deep neural network model (SDAE-BPR) based on Stack Denoising Auto-Encoder and Bayesian Personalized Ranking for the problem of accurate product recommendation. Their results shown high accuracy of parameters estimation and the efficiency of model training. Moreover, authors in [24] introduced a novel approach for service multi-label recommendation using deep neural networks as well. The difference between this research and the previous one lies on the fact that the approach not only satisfy the demands of service multi-label recommendation, but also provide the importance with an ordered label ranking. Moving forward, Liang et al. presented a graph-based approach [25] to automatically assign tags to unlabeled API services by exploiting both graph structure information and semantic similarity. This work has been a basis to the current research, as NSs can also be represented in form of graphs. Taking into consideration the background presented in the previous paragraphs, more effective decision support mechanisms are required in order to recommend to test developers the necessary VNF and NS tests that are of importance. The latter would allow NS developers, V&V test developers, and systems integrators, the ability to provide high quality services, since the targeted tests will reply to the specific requirements of network operators.

Architecture Overview
SONATA (powered by 5GTANG) is a 5GPPP Phase2 Innovation Action that enables the flexible programmability of 5G networks with a) an NFV-enabled Service Development Kit (SDK), b) a Catalogue with advanced validation and verification mechanisms for VNFs/Network Services qualification (including 3rd party contributions) and c) a modular Service Platform with an innovative orchestrator so it can bridge the gap between business needs and network operational management systems.

Software Development Kit (SDK)
The SDK aims at providing the developer with a powerful set of tools to develop, examine and evaluate NFV-based Network Services. Thus, it includes tools and mechanisms for the unified development of the NSs, namely from their creation and validation, until their final packaging and advertisement to the NFV Marketplace. Thus, the latter comprises the key storage point of publishing the emerging services directly from the developer to the diverse stakeholders. Consequently, NFV Marketplace conceptualizes the idea of exploiting existed web services and providing inspiration from the hosted developments. Developers can rapidly initiate projects of Network Services on-the-fly, either by fetching already-advertised Network Services from the NFV Marketplace, or by commencing new developments. The ability of optionally enabling easy re-use of existing components comprises a crucial advantage, which can be blended into the overall service under creation [26].

Validation and Verification Framework
The V&V framework servers the validation and verification of the developed network services [27]. Via the V&V platform, the different stakeholders are able to validate and qualify the reliability of their produced designs prior to release. The advent of the NFV Marketplace constitutes a paramount lever in the V&V ecosystem as it integrates the entire test lifecycle of the VNFs/NS from the execution of the tests to the storage of the results. As it is obvious, the test process demands a plethora of information from low level metrics, logging, system reporting, all of which are included in the NFV Marketplace. The inclusion of the concrete information in the NFV Marketplace provides a holistic insight to the end-user and propels the dynamic nature of the NFV Marketplace for the optimum composition and proposal of innovative test scenarios to the end-user.

Service Platform (SP)
The SP comprises a set of components that provide a service delivery architecture. In the SP, one of the core components of the 5G ecosystem is the flexible MANagement and Orchestration (MANO) framework [28], responsible for the orchestration features and the management of VNFs/NSs. Along with the MANO, several other key enablers coexist as introduced in the 5G environment, such as the Slice [29,30], Policy [31] and the SLA management [32]. The components of the SP produce an abounding amount of information for each respective NS. The MANO framework exports relative information with resource allocation of the instantiated VNFs/NS across the infrastructure. The positioning of the NFV Marketplace comprises the centralized storage entity of all the components of the SP from the necessary files for the deployment of VNFs/NS to the corresponding business rules. This type of information is rendered as profitable knowledge and allows the efficient characterization of the VNFs/NS through their enrichment with valuable metadata in the Catalogue.

Catalogue
The Catalogue is an instrumental component in the SONATA (powered by 5GTANGO) environment present in different parts of it (a common building block with the V&V). Primarily, it hosts the different descriptors of the 5GTANGO packages. Since SONATA (powered by 5GTANGO) aims at a multi-platform environment, it enables the NS developers to orchestrate and deploy their NSs using different Service Platforms. In this context the 5GTANGO Catalogue is being adapted to support the storing and retrieval of new packages, that are "marked" like ONAP [33], OSM [34] or 5GTANGO packages. Moreover, the Catalogue is aligned with the principle of persistent storage by extending the hosted VNFDs and NSDs with valuable fields for successful data integration, accuracy in the format of the document, confirmed time of creation, etc. In this way, it enables the development of enhanced operations for Creating, Retrieving, Updating and Deleting (CRUD) descriptors inside it, while reassures the correct data format of the stored documents. Going beyond a conventional data storage, the presented 5GTANGO Catalogue provides intelligent functionalities in this 5G environment. Since the types of information vary, one of the requirements, satisfied by the Catalogue, is the full-text search capabilities in structure-agnostic documents. Since the schema of the diverse is variable, the Catalogue provides searching capabilities without the necessity of indexes. Thus, it provides seamless retrieval abilities in deep-hierarchical machine-readable document structures. Furthermore, besides the plain NoSQL document store for the diverse descriptors, Catalogue provides a scalable file system for hosting the artifact files, required for the instantiation life cycle of the NSs. Finally, Catalogue provides a set of endpoints where the CRUD methods are supported for the different descriptors of the project.

Positioning of the DSM in the V&V Framework
The decision support mechanism constitutes a vital component in the V&V platform of a 5G environment, implementing methods for suggesting relevant items to the users (i.e. items being tests to execute). This section provides the architectural description of the DSM with the underlying recommendation engine, its consolidation in the 5G environment, in terms of interacting with other software components.

Fig. 1. Positioning of the DSM in the V&V
The emergence of the rapid growth of the available information became clear to its users with the massive advancement of the Internet. RSs has been exploring the internet with the goal of integrating contextual data and offering practical feedback to the end user. While the RSs research field has skyrocketed in a number of domains, there is a steady, but sluggish, interest in applying RSs in 5G environments by defining them in network management applications [35]. Telecom networks generate massive amounts of data monitoring through the advent of the virtualized era, consisting of observations on network failures, configuration, accounting, performance, and safety. E-stream also included a predictive, automated recommendation for network management with surprising results due to the increasing complexity of the networks, associated with specific business level constraints [36]. Through manipulating deep data streams and efficiently applying dimensional reduction techniques, E-stream was focused on recommending behavior with four specific related variables, namely context, audience, existing responses, and validation. The main factor of these recommendations was the appropriate trading policies to enforce the recommendation module and to propel the actions necessary to get the best price per VNF. However, the absence of RSs in 5G telecom systems is crucial in the tacit use of the multiple element QoS/QoE metrics, such as monitoring systems, regulation and SLA modules, etc. RSs approaches and methods therefore form a subset of tools to be thoroughly examined.
In the case of our implementation, a recommendation system which is integrated in the V&V of the SONATA (powered by 5GTANGO) framework, is running at the background "listening" to activities such as choose of a test by a user, upload of a new test package etc. To elaborate more, the most worth mentioned activity is the upload of a new test package. A callback request is arriving at the recommendation engine when a test package is successfully uploaded to the catalogue. Then, the recommendation engine extracts the testing tags from the package descriptor, as well as the owner/user of the package and stores that information in an inside database. At a next step, using the prediction techniques described in section 4.2 and 4.3 the recommendation model is calculated and its ready to provide test recommendations at the next execution of a test case. As it is depicted in Fig.1, the recommendation engine is integrated in the V&V platform, which is communicating with the SP in order to deploy the service and run the recommended test cases. Among others, the SP to execute the test is chosen based on what the user have chosen. Service package can be either OSM, ONAP or SONATA. Thus, based on the kind of the service package, the corresponding SP is automatically chosen.

Overview
The purpose of the recommendation engine is to automate the process of the test selection, providing some recommendation preferences based on the user's previous activity. The recommendation engine uses singular-value decomposition (SVD) methods that are based on collecting and analyzing large amounts of information on users' behaviors, activities or preferences and predict what users will prefer based on their similarity to other users. The main capabilities of the recommendation engine are: • Retrieve the mechanism's health.
• Retrieve the items' "testing tags", the recommendation engine is trained for.
• Retrieve the users, the recommendation engine is trained for.
• Add a new user-item pair based on the uploaded package to the Catalogue.
• Add a new user-item pair based on what test a user has executed.
• Get the top n recommendations for a user.
• Delete a user along with the associate activity.
Aligned with the escalating development of the NFV landscape, the number of services is increasingly overwhelming in the Catalogue. The publishing of the services from the various stakeholders yields a plethora of VNFs/NSs selections to the end-user in the development phase. Therefore, the infiltration, prioritization and delivery of optimum information about VNF selections is considered necessary. The process of optimum selection of VNFs/NSs by the operators is no exception and there is a clear need to facilitate their decision-making process by providing recommendations that match their preferences and needs. The proposed DSM is based in a Collaborative Filtering techique, which makes recommendations based on how users (i.e. test developers) rated items (i.e. test cases) in the past -not based on the items themselves. Specifically, the implemented collaborative-filtering-based recommendation engine is a recommendation system that has no knowledge of the actual item (i.e. test case) it is recommending. On the contrary, it only knows how other users (i.e. test developers) rated the item, as depicted in Figure 2.

Collaborative Filtering
Collaborative Filtering methods are based on collecting and analyzing a large amount of information on users' behaviors, activities or preferences and predicting what users will like based on their similarity to other users. A key advantage of the collaborative filtering approach is that it does not rely on machine analyzable content and therefore it is capable of accurately recommending complex items such as tests without requiring an "understanding" of the item itself. Many algorithms have been used in measuring user similarity or item similarity in recommender systems. For example, singular-value decomposition (SVD) approach. Collaborative filtering is based on the assumption that people who agreed in the past will agree in the future, and that they will like similar kinds of items as they liked in the past.

Singular Value Decomposition
Collaborative Singular-Value Decomposition (SVD) In linear algebra, the singularvalue decomposition (SVD) is a factorization of a real or complex matrix. It is the generalization of the eigen decomposition of a positive semidefinite normal matrix (for example, a symmetric matrix with positive eigenvalues) to any m x n matrix via an extension of the polar decomposition. It has many useful applications in signal processing and statistics.

Continuous Optimization
Upon now, the conventional test planning of the NSs is utterly based on the developer. The continuous optimization service of the recommendation engine introduces dynamicity and adaptability in the operation of the V&V framework. The continuous optimization service it comes to fulfill the integration of V&V platform in the 5G infrastructure. The DSM in collaboration with the Catalogue, target in the assessment of the test performances while considering the actual performance of the deployed VNFs/NS in the SP. Potential differences observed in the performance of the production and the validation environment comprises a basis for extending the strain of the VNFs/NS through the proposal of new test cases to the V&V framework. By means of proposing new test plans, the developer is equipped with innovative tools of improving the design of the VNFs/NSs. The test suggestions will not include only predefined script configurations, but also produce innovative tests to explore potential performance bottlenecks. For these reasons, monitoring information both from the test (i.e validation and verification) and the production infrastructure are collected, along with previous test configurations parameters, that are stored into the Catalogue. After the execution of the new test configurations and the gathering of the corresponding data, the mechanism will propose (i.e. recommend) new V&V tests to be triggered to the V&V framework and thus causing a feedback loop. The activity of the V&V developer is recorded, in the continuous optimization service. Based on this activity, the recommendation engine of the DSM proposes unprecedented test scenarios, according to the test developer history in terms of NS instantiations, corresponding test selections and results.

Conclusions
The current paper presented a Decision Support Mechanism built upon available Network Services, in order to recommend to the test developers, the most relevant tests, aiming to validate and verify the services in the best possible way. The proposed recommendation system uses collaborative filtering taking into consideration the end users' historical preferences on a set of items (i.e. NSs and tests). The f sparsity and scalability issues were tried to be solved using Matrix Factorization, with the Singular Value Decomposition algorithm. It is worth mentioning that collaborative filtering provides significant predictive power for recommender systems, while at the same time requiring the least data. But, in some specific situations, it has some drawbacks, such as the problem with the "cold start". The disadvantages of using collaborative filtering are likely to be solved using a more complete and large set of data. As a future work, we aim to use a more complex dataset, enriched also with a set of metadata, so that we could combine content-based filtering by adding some explicable keyword dimensions, and at the same time be able to consider the trade-offs between model -computational complexity and performance improvement effectiveness.