Deliverable D7.1 Evaluation methodology

The 5G-TOURS project aims at deploying three full end-to-end trials involving real end-users (volunteers who consent to participate) and vertical operational services in three different European cities (Turin, Rennes, and Athens). On the three pilot sites1 , 13 use cases related with the themes of the touristic city (5 use cases), the safe city (4 use cases) and the mobility-efficient city (4 use cases) will be deployed. The ultimate goal of this approach is to trial them in a real environment by continuously collecting network, service and vertical KPIs and evaluate them against a set of predefined vertical-oriented criteria. Toward this direction, the definition and design of a clear evaluation methodology that can guide the pilot sites throughout the whole evaluation process is essential. This deliverable is the first document produced by WP7 - System integration and evaluation, which focuses to deliver the integrated 5G-TOURS ecosystem that will allow for the realisation of pilots in all three nodes and drive the evaluation of the results of the trials. It presents the basic version of the 5G-TOURS evaluation methodology which will guide the pilot sites during all the steps of KPI validation process including: pilot test case design and definition, test case execution, data collection, data analysis and finally evaluation against predefined KPI targets. It provides the basic building blocks for the 5G-TOURS evaluation methodology, dealing with the evaluation of the network KPIs: latency, throughput, reliability, density, mobility, coverage, slice deployment time, security and location accuracy. The first step toward the evaluation is to understand what the Vertical needs concerning trial execution in 5GTOURS pilot sites and design a set of pilot tests. Although some tools exist related to test specification, normally these are more focused on making the life of the test design technician easier rather than on permitting a third party to express particular needs. Therefore, in order to cope with the aforementioned complexity and fill this gap, in 5G TOURS we will design a well-defined evaluation methodology in order to map the vertical requirements in an efficient way into the set of pilot tests. Then, the next step is the actual execution of the pilot tests on the three pilot sites. This step includes the preparation of the sites in terms of deploying the appropriate technologies, deploying the required network functionalities and also prepare the environments for the test execution. Then, the pilot tests are executed and the required metrics are collected. Finally, the collected metrics are analysed and evaluated against a set of predefined criteria. The roadmap toward the final 5G-TOURS evaluation methodology includes the extension of the basic version presented in this document by evaluating the level of satisfaction of end-users and verticals players with the use cases deployed. This includes users' Quality of Experience (QoE) as well as the feedback from the vertical players on how the technology provided can improve their business' operations. The evaluation of end-users satisfaction will deal with QoE metrics derived from network KPIs as well as QoE metrics derived from Mean Opinion Score (MOS) approaches. The second step includes the interaction with T3.5 and will provide the final evaluation methodology fine-tuned to the network capabilities available in the pilot sites (e.g. MEC, SDN, MANO). The 5G-TOURS evaluation methodology presented in the document comprised two parts: • a general part that can be applied to all the project use cases; • a specialised part that defines evaluation procedures that are specific to some use cases and involve particular KPIs. The latter is useful firstly because only a subset of all the project KPIs is critical per each use case and secondly in order to capture the different service requirements and constraints per use case which may need particular network characteristics. In this way, the 5G-TOURS evaluation methodology is both generic (in order to support all the use cases of the project) and at the same time extendable (in order to be customised to support the special needs of each use case).