Kibana, Grafana and Zeppelin on Monitoring data

Project Specification

The goal of this project is to investigate different solutions and develop some typical monitoring displays, such as the general IT Department overviews and service-specific dashboards of data provided by the IT groups and stored in the IT monitoring repository (ElasticSearch). As a part of the project, the current IT host monitoring dashboard and the Data Centre Overview where migrated and ported from Kibana 3 to Grafana and Kibana 4 in order to study features, performance and improvements of these displays. The second part of the project is related to the report visualization of the ATLAS scrutiny report that was implemented in Apache Zeppelin. The last part of the project is related to the porting of the Apache PIG scripts to the Apache Spark computational framework in order to accelerate the generation of the aggregated summaries and to have direct Hadoop access that would be beneficial to any future project development in the report automation.

Abstract

The CERN/IT monitoring team handles every day millions of monitoring events from the CERN data centers and the WLCG sites. Today this data is provided to users through a number of different tools and dashboards. This project aims at exploring new visualization techniques to provide enhanced dashboards. Using the data being stored in ElasticSearch and HDFS this project relies on tools like Kibana, Grafana, and Zeppelin to develop new dashboards relevant for CERN and the LHC Experiments.