TOAR Ozone Data Processing Pipeline for Transformer-Based Forecasting

To download timeseries IDs for all stations

01_download_toar_data.py

11522245

1.fine_station_codes.txt

1. All raw timeseries IDs
2. All timeseries of one station merged together

These can be regenerated by downloading from the API using provided code hence not provided in related repo

To Pivot timeseries info that is one row each stationwise as one column

02_pivot_toar_data.py (also created a serial snippet to explain pivot_simple.py)

11526798

1. all stationWiseCsv files

1. Corresponding Pivoted csvs in provided output paths

To merge all stations into one file to be able to merge with ERA5 (to use nox where exists to fill no and no2 and check station codes and missing values)

03_merge_toar_csvs.py (had equivalent snippets to download ERA5 using CDS API and merging all monthly data into one netcdf grid file with datetime, lat, lon as coordinates but got deleted)

115237325

1. Directory of all Pivoted csv files

1. file saved with nox column dropped after using for no/no2 filling e.g. 359_stations_merged.csv

To merge ERA5 with TOAR

04_merge_toar_era5.py

11534521

1.359_stations_merged.csv
2.Stations_with_Nearest_Grid_Points.csv
3.era5_merged_common_grid.nc

1. With lat and lon mapped from ERA5:
359_stations_updated.csv
2. The merged final file incl. both TOAR and ERA5:
merged_with_era5.csv

To generate samples and finetune by loading the 60 epoch pretrained checkpoints

finetune_urban.py

11504196 (2 weeks ago)
11535024 (Yesterday rerun)

1. merged_with_era5.csv
2.clean_processed_urban_ft_data.csv
3. Checkpoint files