CLIM4cities: D2 Benchmark Report (BR)

This document corresponds to the final release of the CLIM4cities Benchmark Report (BR), deliverable D2 of the Work Package (WP) 2000, which pertains to the results obtained from the Exploratory Data Analysis (EDA) of the compiled and collocated ML model database delivered by WP3000¹, and the development and intercomparison of the alternative air temperature [T2m] and Land Surface Temperature [LST] data-driven model formulations. While a lot of today’s emphasis is given to Machine Learning/Artificial Intelligence (ML/AI) sizes, hyperparameters, and optimisation, a successful predictive model must surpass the challenge of generalisation, particularly when the goal is to deploy such a model into an operational environment and targeting fast time-changing response variables. In what concerns weather and climate predictions, official systems rely on physics-based models which have long been proven as a tool for reliable forecasting. Recently, ML and AI are gaining traction for weather and climate, by showcasing that some pilot models have been able to replicate complex four-dimensional processes with accurate results, leading major ‘power houses’ of the global weather prediction landscape to plan ahead for the ML uptake. However, the major concern regarding this topic is the lack of transparency and explainability of ‘black box’ models, hindering our ability to understand whether it learns the underlying physics of the weather and climate processes, reducing the risk of overfitting or model hallucinations. A way to overcome such insecurities is to ensure that a thorough EDA is conducted in tandem with the ML model development, with its main goals to (i) understand the structure of the ML database and its variables, (ii) ensure the quality of the data (e.g., outliers, physically implausible values, in close cooperation with WP3000) and (iii) disclose existing relationships between variables.

In addition, the ML/AI model development stages reported within this deliverable require domain knowledge and critical reasoning to distinguish existing from spurious relationships, or to flag erroneous data points that should be withdrawn from the ML model database before the model development routine is conducted. From our understanding, even if fine-tuning and different model architectures can offer improvements to the results, its success and accuracy performance is dependent on the quality and meaningfulness of the input data, thus a process of critical reasoning over the structure and type of data features (response variable and predictors) in each step of the process. In D2, we offer an overview of this work, namely a description of the accomplished tasks, as follows:

• WP2100 – ML Database Exploratory Data Analysis [LST and T2m], which is carried out in parallel with WP3000 Non-EO and GEO Data Collection.
• WP2200 – ECV 1 [LST]: ML Development and Benchmark – which includes the train/fine-tuning development routine of at least two (2) ML methods for the Land Surface Temperature [LST] ML model sub-kilometric downscaling.
• WP2300 – ECV 2 [T2m]: ML Development and Benchmark – which includes the train/fine-tuning development routine of at least two (2) ML methods for the Air Temperature [T2m] ML model sub-kilometric downscaling.
• WP2400 – ML Development and Validation [T2m] in additional FUAs – which includes the further developments made fur expanding the model with CERRA and NDVI to new locations i Europe

Considering the CLIM4cities project's revised timeframe, following the Contract Change Notice (CCN) extension, this report can be seen as a living document, describing the pros and cons of each data-driven and ML method currently being explored, particularly benchmarking their key performance indicators (KPIs) of accuracy, but also computational burden. In this report, CLIM4cities outlines the improvement made to attain the best fitted LST and T2m prediction models, to understand how well they detect extreme values in each of the two essential climate variables (ECV), and discuss the results within the scope of approaches reviewed in the State of the Art (SOTA) analysis, in the CLIM4cities Requirements Baseline (RB) deliverable (D1, from WP1000). In the CCN work, we have explored how to take advantage of satellite-based variables and indicators to further improve the T2m model, particularly by introducing vegetation indices that depict the seasonal phenology changes and how these interfere with near-surface air temperatures in urbanised locations. For this we have: (i) replaced DANRA with a European reanalysis dataset, CERRA, (ii) extend the number of FUAs and countries in which the ML model is replicated , and (iii) introduce further satellite predictors in the downscaling process that depict the vegetation seasonality (e.g., NDVI). The regions include Sevilla (ES), Athens (Greece). Copenhagen (DK) and Lisbon (PT). The goal is to scale the solutions further, to prove that the concept works in varied sets of conditions, and also to promote the uptake of such a solution more broadly in Europe.