The LEXIS Distributed Data Infrastructure: Demonstrator System

The LEXIS Distributed Data Infrastructure: Demonstrator System
by: LEXIS Project and Work Package 3 Team

Project Lead: IT4Innovations National Supercomputing Centre (Czech Republic)
Work Package 3 Lead: Leibniz Supercomputing Centre (LRZ, Garching b. M., Germany)

The enormous amounts of data generated in modern industry, business and science pose a significant challenge to those extracting actionable intelligence from data, using various filtering and analysis techniques. In this "Big Data" setting, the LEXIS project (Large-scale EXecution for Industry & Society) provides a user-friendly portal and platform for optimised execution of mixed Cloud-HPC (HPC: High-Performance Computing) workflows. The system will rely on advanced, distributed orchestration solutions (Bull Ystia Orchestrator, based on TOSCA and Alien4Cloud technologies), the High-End Application Execution Middleware HEAppE, and new hardware capabilities for maximizing efficiency in data processing, analysis and transfer (e.g. Burst Buffers with GPU- and FPGA-based data reprocessing).

LEXIS handles computation tasks and data from three Pilots, based on representative and demanding HPC/Cloud-Computing use cases in Industry and Science: i) compute-/data-intensive and time-consuming simulations of turbo-machinery and gearbox systems in Aeronautics, ii) Earthquake and Tsunami simulations which are accelerated to enable accurate real-time analysis, and iii) Weather and Climate HPC simulations where massive amounts of in situ data are assimilated to improve forecasts.

Here, we introduce and show a demonstrator of the LEXIS Distributed Data Infrastructure (DDI), the core data back-end of the LEXIS project. The DDI provides a unified "File Space" for LEXIS, across the participating sites and computing centres. Based on iRODS (irods.org) and EUDAT-B2SAFE (eudat.eu), it will ensure reliable and efficient access to large datasets in the Terabyte range and beyond. We have prepared virtual machine templates for the LEXIS Cloud resources which implement a Demonstrator of the LEXIS DDI. The system, once instantiated, consists of two iRODS-iCAT (provider) servers, representing the LRZ and IT4I iRODS zones, and of two client machines for access to the distributed data management system. Thus, interested colleagues can explore the possibilities offered by this system on an "own" demonstrator instance. Please contact us at info[at]lexis-project.eu if you are interested.