NCAR/miles-credit: v2024.1.0

Our first public release of MILES CREDIT.

What's Changed

• add helper scripts to gather global data and scaling params by @WillyChap in https://github.com/NCAR/miles-credit/pull/1
• Updated dependencies and added scaler application by @djgagne in https://github.com/NCAR/miles-credit/pull/2
• Initial update of multi-step training by @jsschreck in https://github.com/NCAR/miles-credit/pull/3
• Adding PBC to crossformer model by @jsschreck in https://github.com/NCAR/miles-credit/pull/6
• Path fixing by @jsschreck in https://github.com/NCAR/miles-credit/pull/8
• reworked spectralLoss2D to be compatible with new models by @dkimpara in https://github.com/NCAR/miles-credit/pull/7
• fixed spectralLoss2D when using no lat weights, also removed deprecated code by @dkimpara in https://github.com/NCAR/miles-credit/pull/9
• Updating few bugs in model classes. Using units now in predict.py by @jsschreck in https://github.com/NCAR/miles-credit/pull/11
• zarrify.py script update by @sethmcg in https://github.com/NCAR/miles-credit/pull/4
• wrote PSDLoss, benchmarked values, added option fields to configs by @dkimpara in https://github.com/NCAR/miles-credit/pull/14
• Some quick improvments of draw_forecast within predict.py by @yingkaisha in https://github.com/NCAR/miles-credit/pull/16
• using os.path.expandvars in train.py,predict.py to enable generic save locs by @dkimpara in https://github.com/NCAR/miles-credit/pull/17
• Update predict.py, visualization_tools.py, and config file options by @yingkaisha in https://github.com/NCAR/miles-credit/pull/20
• Updated FSDP checkpointing by @jsschreck in https://github.com/NCAR/miles-credit/pull/21
• polar and laplacian diffusion filter module class by @WillyChap in https://github.com/NCAR/miles-credit/pull/23
• predict.py revamp - viz config options, async image generation, xarray creation, laplacian filtering, CPU-only runtime compatible by @dkimpara in https://github.com/NCAR/miles-credit/pull/27
• Modifications of predict.py by @jsschreck in https://github.com/NCAR/miles-credit/pull/28
• predict.py - reworked xr save format, using logger by @dkimpara in https://github.com/NCAR/miles-credit/pull/29
• Fixing FSDP revert; adding multi-step trajectory trainer by @jsschreck in https://github.com/NCAR/miles-credit/pull/31
• Quantile static by @WillyChap in https://github.com/NCAR/miles-credit/pull/32
• Updates to xformer model class and xformer configurations by @jsschreck in https://github.com/NCAR/miles-credit/pull/34
• Model base class by @dkimpara in https://github.com/NCAR/miles-credit/pull/35
• FuXi bug fix + Colorbar adjustments for visualization_tools by @yingkaisha in https://github.com/NCAR/miles-credit/pull/36
• TOA var and Quantile scaling, by @WillyChap in https://github.com/NCAR/miles-credit/pull/37
• Predict.py now supports FSDP by @jsschreck in https://github.com/NCAR/miles-credit/pull/38
• Adding static inputs by @jsschreck in https://github.com/NCAR/miles-credit/pull/39
• Conus404 data loader by @sethmcg in https://github.com/NCAR/miles-credit/pull/22
• adding transform/inverse for bscaler data by @WillyChap in https://github.com/NCAR/miles-credit/pull/40
• Addition of CONUS404 data, model, and training scripts by @jsschreck in https://github.com/NCAR/miles-credit/pull/41
• Updated FSDP-related bugs by @jsschreck in https://github.com/NCAR/miles-credit/pull/42
• edge case fix by @WillyChap in https://github.com/NCAR/miles-credit/pull/44
• Graph residual transformer with sparse edge calculation by @djgagne in https://github.com/NCAR/miles-credit/pull/26
• KE and spectrum visualization diagnostics + data conversions by @dkimpara in https://github.com/NCAR/miles-credit/pull/43
• Adding the replay buffer code as example by @jsschreck in https://github.com/NCAR/miles-credit/pull/45
• Adding Climate run capabilites by @WillyChap in https://github.com/NCAR/miles-credit/pull/46
• Fuxi update by @jsschreck in https://github.com/NCAR/miles-credit/pull/47
• Start of major refactoring of code base by @jsschreck in https://github.com/NCAR/miles-credit/pull/48
• TOA data type conversion and input tensor shape comment by @kanz76 in https://github.com/NCAR/miles-credit/pull/51
• FuXi model updates and data pipeline initial work by @yingkaisha in https://github.com/NCAR/miles-credit/pull/52
• Added Swin transformer model + new rollout script by @jsschreck in https://github.com/NCAR/miles-credit/pull/53
• The new data pipeline by @yingkaisha in https://github.com/NCAR/miles-credit/pull/55
• Data pipeline update with minor bugfix and documentation by @yingkaisha in https://github.com/NCAR/miles-credit/pull/58
• NetCDF metadata, docs, environment fixes by @djgagne in https://github.com/NCAR/miles-credit/pull/56
• Conus404 transforms by @sethmcg in https://github.com/NCAR/miles-credit/pull/54
• add 6hourly cached option by @WillyChap in https://github.com/NCAR/miles-credit/pull/57
• Emergency bug fix for the new roll-out and metadata loader by @yingkaisha in https://github.com/NCAR/miles-credit/pull/60
• update yaml to work by @WillyChap in https://github.com/NCAR/miles-credit/pull/61
• Add 6 hourly run configs with the new data pipeline + minor bugfix by @yingkaisha in https://github.com/NCAR/miles-credit/pull/62
• rollout_to_netcdf_new.py that works with the new data pipeline by @yingkaisha in https://github.com/NCAR/miles-credit/pull/64
• Updates to multi-step training a la Brenowitz scheme by @jsschreck in https://github.com/NCAR/miles-credit/pull/63
• Bugfix on rollout_to_netcdf_new.py + adding example.yml by @yingkaisha in https://github.com/NCAR/miles-credit/pull/67
• Memory optimization for one_shot=True + minor corrections for the example.yml by @yingkaisha in https://github.com/NCAR/miles-credit/pull/69
• Initialization of credit.trainers by @jsschreck in https://github.com/NCAR/miles-credit/pull/65
• Updates to the multi-step training code by @jsschreck in https://github.com/NCAR/miles-credit/pull/70
• new lat weighting by @WillyChap in https://github.com/NCAR/miles-credit/pull/66
• Add dynamic_forcing_variables into the data workflow by @yingkaisha in https://github.com/NCAR/miles-credit/pull/73
• Initial code for the Graph Residual Transformer + GRU model by @kanz76 in https://github.com/NCAR/miles-credit/pull/72
• Added reload_epoch option to the trainers by @jsschreck in https://github.com/NCAR/miles-credit/pull/76
• minor adjustments on credit.model + update configs by @yingkaisha in https://github.com/NCAR/miles-credit/pull/74
• Bugfix on model checkpointing and epoch update by @yingkaisha in https://github.com/NCAR/miles-credit/pull/77
• New solar processing and bridgescaler state scaler by @djgagne in https://github.com/NCAR/miles-credit/pull/78
• Updating the swin transformer by @jsschreck in https://github.com/NCAR/miles-credit/pull/80
• Adding parser and input data checks for config.yml + some minor fixes by @yingkaisha in https://github.com/NCAR/miles-credit/pull/79
• Model check-pointing and early-stopping fixes for base_trainer by @yingkaisha in https://github.com/NCAR/miles-credit/pull/81
• Bug fixes and few updates with variable weights by @jsschreck in https://github.com/NCAR/miles-credit/pull/82
• Small bugfix on parsing one_shot=False by @yingkaisha in https://github.com/NCAR/miles-credit/pull/88
• Updates to multi-step training by @jsschreck in https://github.com/NCAR/miles-credit/pull/87
• Adding arXiv paper production run configs + documentation by @yingkaisha in https://github.com/NCAR/miles-credit/pull/89
• [In Progress]: PBS Script and some minor improvements. by @negin513 in https://github.com/NCAR/miles-credit/pull/90
• few typos by @negin513 in https://github.com/NCAR/miles-credit/pull/92
• Fixes to new MPI launch; linting; reorganizing distributed by @jsschreck in https://github.com/NCAR/miles-credit/pull/93
• add lev functionality by @WillyChap in https://github.com/NCAR/miles-credit/pull/84
• Adding NVIDIA-makani scheme for multi-step training / rollout script for computing metrics on forecasts by @jsschreck in https://github.com/NCAR/miles-credit/pull/91
• multistep configurations + optimized cached dataset workflow + old script clean-up by @yingkaisha in https://github.com/NCAR/miles-credit/pull/94
• Updated learning rate scheduler and metrics handling by @jsschreck in https://github.com/NCAR/miles-credit/pull/95
• Notebooks and new github actions workflow by @djgagne in https://github.com/NCAR/miles-credit/pull/97
• Updated README with credit-derecho installation instructions added by @jsschreck in https://github.com/NCAR/miles-credit/pull/99
• Updated MPI support rollout_metrics by @jsschreck in https://github.com/NCAR/miles-credit/pull/100
• Emergency fix on rollout_to_netcdf.py and rollout_metrics by @yingkaisha in https://github.com/NCAR/miles-credit/pull/104
• Bugfix on trainerERA5_multistep_grad_accum.py by @yingkaisha in https://github.com/NCAR/miles-credit/pull/106
• Infrastructure for post-processing blocks (e.g. SKEBS, mass correction, laplacian filtering) by @dkimpara in https://github.com/NCAR/miles-credit/pull/105
• fixing environment_[device].yml specs by @dkimpara in https://github.com/NCAR/miles-credit/pull/108
• Major credit.postblock update + major credit.physics_core update + major bugfix + minor code optimization and cleaning by @yingkaisha in https://github.com/NCAR/miles-credit/pull/86
• Bugfix on the use of diagnostic variables in trainerERA5_multistep_grad_accum.py by @yingkaisha in https://github.com/NCAR/miles-credit/pull/111
• Fix NCLL_NET_GDR_LEVEL environment variable in credit.pbs.py by @kanz76 in https://github.com/NCAR/miles-credit/pull/115
• PHB does not work. Revert to PBH by @kanz76 in https://github.com/NCAR/miles-credit/pull/118
• credit.postblock major updates on GlobalMassFixer and GlobalEnergyFixer by @yingkaisha in https://github.com/NCAR/miles-credit/pull/110
• update unet init and call in models/init.py by @dkimpara in https://github.com/NCAR/miles-credit/pull/119
• updating the periodic padding function to roll then reflect lats by @WillyChap in https://github.com/NCAR/miles-credit/pull/117
• edits to pbs.py to improve usability by @dkimpara in https://github.com/NCAR/miles-credit/pull/114
• URGENT bugfix: removed cwd keyword from subprocess.popen by @dkimpara in https://github.com/NCAR/miles-credit/pull/124
• Add Python unit testing for credit.models.Fuxi and credit.boundary_padding by @yingkaisha in https://github.com/NCAR/miles-credit/pull/121
• Pressure interpolation and a bunch of formatting changes by @djgagne in https://github.com/NCAR/miles-credit/pull/125
• removed cwd from popen again. by @dkimpara in https://github.com/NCAR/miles-credit/pull/127
• Bugfix on the production of diagnostic variables in rollout_to_netcdf.py + credit.postblock update + other minor bugfix by @yingkaisha in https://github.com/NCAR/miles-credit/pull/120
• Add torch.clamp to trainers + add drop_path in FuXi + bugfix FuXi when use_spectral_norm: False + minor updates on credit.parser by @yingkaisha in https://github.com/NCAR/miles-credit/pull/130
• Fix mpi in rollout_to_netcdf.py by @kanz76 in https://github.com/NCAR/miles-credit/pull/132
• Bugifx on random seed inconsistencies and keyword grad_max_norm by @yingkaisha in https://github.com/NCAR/miles-credit/pull/133
• Version changes by @djgagne in https://github.com/NCAR/miles-credit/pull/136

New Contributors

• @WillyChap made their first contribution in https://github.com/NCAR/miles-credit/pull/1
• @djgagne made their first contribution in https://github.com/NCAR/miles-credit/pull/2
• @jsschreck made their first contribution in https://github.com/NCAR/miles-credit/pull/3
• @dkimpara made their first contribution in https://github.com/NCAR/miles-credit/pull/7
• @sethmcg made their first contribution in https://github.com/NCAR/miles-credit/pull/4
• @kanz76 made their first contribution in https://github.com/NCAR/miles-credit/pull/51
• @negin513 made their first contribution in https://github.com/NCAR/miles-credit/pull/90

Full Changelog: https://github.com/NCAR/miles-credit/commits/2024.1.0