Processing of the WLCG job monitoring data using ElasticSearch

Abstract

The Worldwide LHC Computing Grid (WLCG) includes more than 170 grid and cloud computing centres in 40 countries. More than 2 million computational jobs are being executed on a daily basis and petabytes of data are transferred between sites. Monitoring the job processing activity of the LHC experiments, over such a huge heterogeneous infrastructure, is really demanding in terms of computation, performance and reliability. Furthermore, the generated job monitoring flow is constantly increasing, which represents another challenge for the monitoring systems.

While existing solutions are traditionally based on Oracle for data storage and processing, recent developments in the SDC monitoring team evaluate different NoSQL solutions for processing large-scale monitoring datasets. Among those solutions is ElasticSearch – an open source distributed real time search and analytics engine. The aim of this project is to prototype the WLCG Job Monitoring applications to store and retrieve data using ElasticSearch.