HiDALGO2 D4.4 Advances in HPDA and AI for Global Challenges

Deliverable D4.4 of the HiDALGO2 project presents a comprehensive account of the significant advances achieved in High-Performance Data Analytics (HPDA) and Artificial Intelligence (AI) since the previous report, D4.3. It reflects the project’s maturation from conceptual and prototypical developments into operational, scalable systems capable of supporting real-world environmental challenges. The document highlights not only methodological innovations but also the establishment of robust infrastructures and reproducible workflows that combine large-scale data processing with advanced AI models across multiple application domains.
The project’s computational ecosystem has evolved into an integrated HPDA-AI platform built on Apache Spark, HDFS, and JupyterHub. This architecture now supports complex analytics workflows with enhanced provenance tracking, deterministic job orchestration, and secure user management, ensuring scalability, reproducibility, and interoperability among HiDALGO2 partners.
Within this technological framework, all five HiDALGO2 pilot domains demonstrate clear progress. The Urban Air Project has moved from an initial abstract concept to an operational, large-scale data analytics pipeline and an under-development AI workflow. Its HPDA component transforms computational fluid dynamics simulations of urban airflow and pollutant transport into actionable compliance assessments aligned with European air quality directives. The new data pipeline can process annual-scale simulations, performing analysis in three different levels of temporal granularity, in under five hours, identifying cases where pollutant concentration exceeds regulatory thresholds. In parallel, an AI-based surrogate emulator is being designed to replicate wind and pollutant dispersion patterns without the need for full-scale simulations – the objective being the development of a full-fledged AI application trained with simulation data, enabling predictive urban air quality management and faster scenario analysis. 
The Urban Buildings pilot now operates a sophisticated data processing pipeline that aggregates detailed solar exposure simulations into city-scale hexagonal maps of parameterizable granularity, using distributed Spark-based computation to efficiently handle multi-gigabyte datasets. On top of this analytical foundation, the AI component has successfully employed graph neural networks to model and predict the impact of new constructions on solar accessibility and daylight availability for a fraction of the available datasets for this pilot. The vision of HiDALGO2 is to utilise the combination of scalable data processing and interpretable AI to provide urban planners with a powerful toolkit to assist their decision-making process for designing energy-efficient and sustainable built environments.
In the Renewable Energy Sources pilot, HPDA and AI methods have been fully integrated into a single operational forecasting framework. Ensemble-based analytics aggregate and weight meteorological simulations to produce reliable climatological insights, while neural networks trained on high-frequency data from a real photovoltaic farm deliver accurate predictions of daily power generation. The workflow bridges weather modelling and renewable energy forecasting, enhancing resilience and planning under uncertain climatic conditions.
The Wildfires pilot successfully combines large-scale HPDA techniques for burn probability estimation with deep learning models that capture the spatiotemporal evolution of fire dynamics. Long short-term memory autoencoders and computer vision-based feature extraction methods have been introduced, enabling the characterization and retrieval of similar wildfire events for rapid response and scenario evaluation. These developments move beyond handcrafted analyses toward automated, data-driven prediction and pattern discovery.
A new pilot domain, Material Transport in Water, marks an important expansion of HiDALGO2’s scope. Using the ChannelFlow-Tools pipeline, a collection of over ten thousand high-resolution lattice Boltzmann simulations has been generated, creating a 20-terabyte dataset suitable for training machine learning applications. On this foundation, a 3D U-Net surrogate model was trained to emulate flow dynamics with remarkable accuracy and computational efficiency. The result is a scalable, AI-driven approach to modelling environmental fluid processes that would otherwise require extensive high-performance simulation time.
Across all HiDALGO2 pilots, the project is targeting the convergence on a unified approach that couples high-performance data infrastructures with intelligent modelling, transforming raw simulation outputs into operational decision-support tools. The integration of distributed analytics, automated data management, and domain-specific AI demonstrates the maturity of the HiDALGO2 framework as a platform for environmental resilience and sustainability research.
In conclusion, D4.4 marks a decisive step forward in realizing HiDALGO2’s vision of harnessing HPC and AI to address global challenges. It consolidates the technical foundations established in the earlier version of this report (D4.3) and transitions them into functional, scalable, and validated workflows. The work reported here not only delivers tangible improvements in computational and analytical capabilities but also establishes a replicable methodology for cross-domain applications. Looking ahead, future activities will focus on expanding dataset coverage, embedding physics-informed AI principles, enhancing workflow efficiency, and incorporating expert feedback to refine predictive accuracy and usability. Collectively, these advancements reinforce HiDALGO2’s contribution to the European effort in developing high-impact, sustainable, and intelligent computational tools for understanding and managing complex global systems.