v2024.1.0

Our first public release of MILES CREDIT.

What's Changed

• add helper scripts to gather global data and scaling params by @WillyChap in #1
• Updated dependencies and added scaler application by @djgagne in #2
• Initial update of multi-step training by @jsschreck in #3
• Adding PBC to crossformer model by @jsschreck in #6
• Path fixing by @jsschreck in #8
• reworked spectralLoss2D to be compatible with new models by @dkimpara in #7
• fixed spectralLoss2D when using no lat weights, also removed deprecated code by @dkimpara in #9
• Updating few bugs in model classes. Using units now in predict.py by @jsschreck in #11
• zarrify.py script update by @sethmcg in #4
• wrote PSDLoss, benchmarked values, added option fields to configs by @dkimpara in #14
• Some quick improvments of draw_forecast within predict.py by @yingkaisha in #16
• using os.path.expandvars in train.py,predict.py to enable generic save locs by @dkimpara in #17
• Update predict.py, visualization_tools.py, and config file options by @yingkaisha in #20
• Updated FSDP checkpointing by @jsschreck in #21
• polar and laplacian diffusion filter module class by @WillyChap in #23
• predict.py revamp - viz config options, async image generation, xarray creation, laplacian filtering, CPU-only runtime compatible by @dkimpara in #27
• Modifications of predict.py by @jsschreck in #28
• predict.py - reworked xr save format, using logger by @dkimpara in #29
• Fixing FSDP revert; adding multi-step trajectory trainer by @jsschreck in #31
• Quantile static by @WillyChap in #32
• Updates to xformer model class and xformer configurations by @jsschreck in #34
• Model base class by @dkimpara in #35
• FuXi bug fix + Colorbar adjustments for visualization_tools by @yingkaisha in #36
• TOA var and Quantile scaling, by @WillyChap in #37
• Predict.py now supports FSDP by @jsschreck in #38
• Adding static inputs by @jsschreck in #39
• Conus404 data loader by @sethmcg in #22
• adding transform/inverse for bscaler data by @WillyChap in #40
• Addition of CONUS404 data, model, and training scripts by @jsschreck in #41
• Updated FSDP-related bugs by @jsschreck in #42
• edge case fix by @WillyChap in #44
• Graph residual transformer with sparse edge calculation by @djgagne in #26
• KE and spectrum visualization diagnostics + data conversions by @dkimpara in #43
• Adding the replay buffer code as example by @jsschreck in #45
• Adding Climate run capabilites by @WillyChap in #46
• Fuxi update by @jsschreck in #47
• Start of major refactoring of code base by @jsschreck in #48
• TOA data type conversion and input tensor shape comment by @kanz76 in #51
• FuXi model updates and data pipeline initial work by @yingkaisha in #52
• Added Swin transformer model + new rollout script by @jsschreck in #53
• The new data pipeline by @yingkaisha in #55
• Data pipeline update with minor bugfix and documentation by @yingkaisha in #58
• NetCDF metadata, docs, environment fixes by @djgagne in #56
• Conus404 transforms by @sethmcg in #54
• add 6hourly cached option by @WillyChap in #57
• Emergency bug fix for the new roll-out and metadata loader by @yingkaisha in #60
• update yaml to work by @WillyChap in #61
• Add 6 hourly run configs with the new data pipeline + minor bugfix by @yingkaisha in #62
• rollout_to_netcdf_new.py that works with the new data pipeline by @yingkaisha in #64
• Updates to multi-step training a la Brenowitz scheme by @jsschreck in #63
• Bugfix on rollout_to_netcdf_new.py + adding example.yml by @yingkaisha in #67
• Memory optimization for one_shot=True + minor corrections for the example.yml by @yingkaisha in #69
• Initialization of credit.trainers by @jsschreck in #65
• Updates to the multi-step training code by @jsschreck in #70
• new lat weighting by @WillyChap in #66
• Add dynamic_forcing_variables into the data workflow by @yingkaisha in #73
• Initial code for the Graph Residual Transformer + GRU model by @kanz76 in #72
• Added reload_epoch option to the trainers by @jsschreck in #76
• minor adjustments on credit.model + update configs by @yingkaisha in #74
• Bugfix on model checkpointing and epoch update by @yingkaisha in #77
• New solar processing and bridgescaler state scaler by @djgagne in #78
• Updating the swin transformer by @jsschreck in #80
• Adding parser and input data checks for config.yml + some minor fixes by @yingkaisha in #79
• Model check-pointing and early-stopping fixes for base_trainer by @yingkaisha in #81
• Bug fixes and few updates with variable weights by @jsschreck in #82
• Small bugfix on parsing one_shot=False by @yingkaisha in #88
• Updates to multi-step training by @jsschreck in #87
• Adding arXiv paper production run configs + documentation by @yingkaisha in #89
• [In Progress]: PBS Script and some minor improvements. by @negin513 in #90
• few typos by @negin513 in #92
• Fixes to new MPI launch; linting; reorganizing distributed by @jsschreck in #93
• add lev functionality by @WillyChap in #84
• Adding NVIDIA-makani scheme for multi-step training / rollout script for computing metrics on forecasts by @jsschreck in #91
• multistep configurations + optimized cached dataset workflow + old script clean-up by @yingkaisha in #94
• Updated learning rate scheduler and metrics handling by @jsschreck in #95
• Notebooks and new github actions workflow by @djgagne in #97
• Updated README with credit-derecho installation instructions added by @jsschreck in #99
• Updated MPI support rollout_metrics by @jsschreck in #100
• Emergency fix on rollout_to_netcdf.py and rollout_metrics by @yingkaisha in #104
• Bugfix on trainerERA5_multistep_grad_accum.py by @yingkaisha in #106
• Infrastructure for post-processing blocks (e.g. SKEBS, mass correction, laplacian filtering) by @dkimpara in #105
• fixing environment_[device].yml specs by @dkimpara in #108
• Major credit.postblock update + major credit.physics_core update + major bugfix + minor code optimization and cleaning by @yingkaisha in #86
• Bugfix on the use of diagnostic variables in trainerERA5_multistep_grad_accum.py by @yingkaisha in #111
• Fix NCLL_NET_GDR_LEVEL environment variable in credit.pbs.py by @kanz76 in #115
• PHB does not work. Revert to PBH by @kanz76 in #118
• credit.postblock major updates on GlobalMassFixer and GlobalEnergyFixer by @yingkaisha in https://github.com/NCAR/miles-credit/...