D3.2 – WP 3 Scientific Report and Prototype Description – Y2

This is the Scientific Report and Prototype Description (Y2), reflecting the work done in the scope of the Data-Driven Infrastructure Management (DDIM) capability of the overall BigDataStack environment. The document describes the DDIM solution as assembled at M23 of the project (i.e. November 2019) in terms of updated design specifications, implementation, integration details, experimentation outcomes and next steps for the highlevel components comprising the DDIM solution: Cluster Management, Dynamic Orchestration, ADS Ranking & Deploy, Triple Monitoring & QoS Evaluation, and InformationDriven Networking. Regarding research results, it focuses on the research conducted to optimize the two components bringing artificial intelligence (AI) to the solution: the ADS Ranking—responsible for ranking and selecting the best application deployment configurations—and the Dynamic Orchestration—in charge of making re-deployment decisions. Both components make use of machine learning (ML) techniques to bring the data-driven aspect to the DDIM capability. The rest of the components complement the above towards the overall data-driven functionality targeted at the level of infrastructure management: Cluster Management and Information-Driven Networking, responsible for the management and monitoring of the infrastructure resources (compute, storage and networking), and the Triple Monitoring & QoS Evaluation, responsible for the monitoring and evaluation of the QoS at different levels of the solution, that is, infrastructure resources, data services and application services. This document concludes with the description of global experimentation outcomes related to the ranking of the performance of grocery product recommendation techniques for the Connected Consumer use case.