# Code and data set from data fusion uncertainty-enabled methods to map street-scale hourly NO2 in Barcelona city: a case study with CALIOPE-Urban v1.0 **Authors & Correspondence** : Álvaro Criado Romero ([alvaro.criado\@bsc.es](mailto:alvaro.criado@bsc.es){.email}), Jan Mateu Armengol ([jan.mateu\@bsc.es](mailto:jan.mateu@bsc.es){.email}) **Last modification**: 22/03/2023 ## 1. Context Universal Kriging is a common geostatistic technique used for spatial interpolation that combines a (multi)linear regression analysis ---with auxiliary variables called *covariates*--- along with a spatial interpolation ---carried out considering the auto-correlated spatial structure of the data---. We have applied this methodology as a post-process of the urban air quality dispersion model CALIOPE-Urban, developed by the Earth Science Department of the Barcelona Supercomputing Center (BSC) (Barcelona, Spain). To implement it, we have used the hourly observational NO2 data from 12 monitoring stations as the principal variable and the CALIOPE-Urban NO2 output as the covariate. In addition, we have studied the added value to incorporate as the second covariate our time-invariant microscale-Land Use Regression (LUR) model, developed by using two different NO2 passive dosimeters campaigns, and 8 predictors (urban geometric variables, simulated vehicular traffic densities, annually-averaged data bi-linearly interpolated from the regional CALIOPE system and the annually-averaged NO2 output of CALIOPE-Urban) through a machine learning approach. Our implementation is a data fusion procedure utilized as a spatial NO2 bias correction technique within an urban area, specifically for the 2019 data collected over Barcelona city. Notice that this code is linked with a journal article: Criado, A., Mateu Armengol, J., Petetin, H., Rodríguez-Rey, D., Benavides, J., Guevara, M., Pérez García- Pando, C., Soret, A. & Jorba, O. (2022). Data fusion uncertainty-enabled methods to map street-scale hourly NO2 in Barcelona city: a case study with CALIOPE-Urban v1. 0. *EGUsphere*, 1-27. For more information, suggestions, or clarifications, please do not hesitate to write an email to the Authors. ## 2. First Steps: understanding the scripts The zip file *UniversalKriging_acriado_jmateu.zip* contains a folder, called *Code (*with a unique sub-folder called *src),* and two archives, *kriging_repository_final.R* and *config_file_final.yml*. The main R script is *kriging_repository_final.R*, which invokes all the R scripts in the *src* folder necessary for its workflow. These scripts are categorized based on their general actions (e.g., *cross_UK1.R*, *stats_doc.R*), or specific actions within subfolders of src (e.g., *LUR*, *obs*). ### 2.1 The configuration file *(config_file_final.yml)* The configuration file is an archive used as a setup structure, meaning that the variables that appear in it can be changed and produce a different output. It is a separate file, so the main advantage is that it can be modified without varying the rest of the scripts. The first step to begin consists of filling it out. Notice that this is the only archive that has to be modified. The user must define the following items: - **UK_mode** : (***UK1*** or ***UK2***) ***UK1*** = UK-DM application (terminology used in the scientific article) ***UK2*** = UK-DM-LUR application (terminology used in the scientific article) This parameter is referred to using or not the microscale-LUR model as the second covariate. If *UK_mode* is *UK1*, only CALIOPE-Urban is used as the covariate. If *UK_mode* is *UK2*, both CALIOPE-Urban and the microscale-LUR model will be the covariates. This has to be written between [ ]. Note that the microscale-LUR model will be used only if the UK_mode is UK2. Be aware that all scripts, folders, and elements are classified into UK1 and UK2 categories. Therefore, we remind the user to link UK1 and UK2 with the UK-DM and UK-DM-LUR methodologies, respectively. - **application** : (all possible combinations using the following items: ***cross*** and ***UK*** ) This is referred to the application of the methodology that we want. This/They has/have to be written between [ ]: 1. ***cross***: hourly Leave-One-Out Cross-Validation, and get the results only at the monitoring stations. 2. ***UK***: Universal Kriging annual correction over the entire domain (the default) and hourly correction over the whole domain for 28/02/2019 (it is specified in the scientific article) - ***plot_images*** : (***TRUE*** or ***FALSE***) If ***TRUE***, plots (maps with the predicted values and the different results explained in the journal article) are generated. - ***evaluation*** : (***TRUE*** or ***FALSE***) If ***TRUE***, statistical evaluation is performed - ***n_cores*** : (a number) It is linked with the computational parallelization carried out using the package multiApply () and prepared in the Barcelona Supercomputing Center environment. The standard number of cores is 16. In case of not using it, set the number as 1. - ***campaigns*** : (***TRUE*** or ***FALSE***) To assess the performance of each dosimeter campaign in the microscale-LUR model and full data-fusion methodology: ***TRUE***; if not: ***FALSE***. See the Appendix A section in the cited journal article. - ***minimum_receptors*** : (***TRUE*** or ***FALSE***) To assess the performance of the microscale-LUR model based on the number of samplers integrated into the LUR: ***TRUE***; if not: ***FALSE***. See the Appendix A section in the cited journal article. ### 2.2 The main script and its explanation *(kriging_repository_final.R)* The *kriging_repository_final.R* script is the main script regarding the application of the methodology and the one that will be submitted. Please, open it to see the code and the following explanation. Notice that the user does not have to change this script; the only part ready to change is the configuration file*.* It is split into different sections: - The first section, ***Config file***, is about setting up the procedure by reading the configuration file. - The ***initial setting*** section considers the R libraries that will be used and the coordinates reference. - In the ***initial setting: directories*** section, the folders' structure is created. About the results, a folder named *Results_user_test* will appear at the same hierarchical level as the *Code* folder. If the folders have already been created, nothing will happen. In this case, the folder containing the actual results linked to the journal article is also attached and named *Results journal article*. - In the ***initial setting: variables*** section: 1. Dates are being set up. 2. The mesh where the bias correction occurs and the observational sites are introduced. The mesh is the same as the CALIOPE-Urban one, with approximately 49000 points. - In the ***observations*** section, a script is used for reading the observations, preparing their format, etc. The observations (monitoring station data) are publicly available through the Catalan Atmospheric Pollution Surveillance Network (XVPCA): [https://mediambient.gencat.cat/es/05_ambits_dactuacio/atmosfera/qualitat_de_laire/vols-saber-que-respires/descarrega-de-dades/](https://mediambient.gencat.cat/es/05_ambits_dactuacio/atmosfera/qualitat_de_laire/vols-saber-que-respires/descarrega-de-dades/%22) - In the ***caliope evaluation and mean*** section, different files regarding the model (CALIOPE-Urban) output at the monitoring stations (caliope evaluation) and the mean annual values (caliope mean) are called. - In the ***variogram total*** section, the variogram and its model are created. The current variogram model uses data from the IDAEA-CSIC dosimeters campaign, so the data cannot be publicly available (contact them if interested in these datasets). The authors of this repository have prepared a test set to keep running the script. - The ***covariates*** section is only used if the ***UK_mode*** is *UK2* since it refers to the microscale-LUR model. Two scripts (*LUR_performance.R* & *LUR_final.R*) are employed to address its expected performance and the basemap, respectively. If the user sets as *TRUE* the parameters *campaigns* and *minimum_receptors* in the configuration file, also an analogy procedure will be carried out based on what is explained in the Appendix section of the cited article. Note that, about the dosimeters campaigns, all folders and information related to the CSIC one are named "dosim2017". For the xAire campaign, as "xAire". - In the ***application*** section, all the applications established by the user are made. If the user sets as *TRUE* the parameters *campaigns* and *minimum_receptors* in the configuration file, also an analogy procedure will be carried out based on what is explained in the Appendix section of the cited article. - In the ***plots*** section, the images are generated if the configuration's parameter *plot_images* is *TRUE*. - In the ***evaluation*** section, some statistical operations are carried out if *evaluation* is *TRUE*. Note that some script parts require data from the IDAEA-CSIC dosimeters campaign. The authors of this repository have prepared a test set to keep running the script. As a general remark, the authors would like to emphasize that the code is functional. However, in cases where specific data is confidential or cannot be publicly released due to external dependencies, fake datasets have been provided to external users. Nonetheless, all journal article results that do not violate these restrictions have been openly published, including the values of the microscale-LUR model basemap (but not its predictors). If you have any questions or concerns, please get in touch with the authors ## 3. Results The folder *Results journal article* contains all the results **not subject to confidentiality.** The folders' structure is the following: **Results journal article** **---\>** **2019** (the year of correction) - **UK1** (UK-DM methodology) - **UK** (results) - **cross** (hourly LOOCV results at the monitoring stations) - **UK_mean** (NO2 annual post-processed kriging values over the whole domain) - **uncertainty** (NO2 annual post-processed kriging sd over the whole domain) - **images** (maps) - **UK2** (UK-DM-LUR methodology) - **UK** (results) - **cross** (hourly LOOCV results at the monitoring stations) o **xAire** (hourly LOOCV results when using only the xAire campaign in the microscale-LUR model at the monitoring stations) o **dosim2017** (hourly LOOCV results when using only the CSI campaign in the microscale-LUR model at the monitoring stations) - **UK_mean** (NO2 annual post-processed kriging values over the whole domain) - **uncertainty** (NO2 annual post-processed kriging sd over the whole domain) - **UK_daily** (NO2 hourly post-processed kriging values for 28/02/2019 over the whole domain) - **images** (maps) - **LUR** - **basemap** (results of the microscale-LUR basemap with and without the correction of the residual) - **performance** (results of the microscale-LUR model's performance) o **xAire** (performance when using only the xAire campaign) o **dosim2017** (performance when using only the CSIC campaign) o **test_receptors** (performance when testing the number of samplers ➤ receptors_140 ... (uniform increments of 50 samplers) ➤ receptors_790 ➤ receptors_844 - **stats_doc** (statistical results and other figures present in the journal article)