Skip to content

v2024.1.0

Latest
Latest
Compare
Choose a tag to compare
@djgagne djgagne released this 10 Dec 14:17
· 10 commits to main since this release
v2024.1.0
a326a6c
This commit was created on github.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.

Our first public release of MILES CREDIT.

What's Changed

  • add helper scripts to gather global data and scaling params by @WillyChap in #1
  • Updated dependencies and added scaler application by @djgagne in #2
  • Initial update of multi-step training by @jsschreck in #3
  • Adding PBC to crossformer model by @jsschreck in #6
  • Path fixing by @jsschreck in #8
  • reworked spectralLoss2D to be compatible with new models by @dkimpara in #7
  • fixed spectralLoss2D when using no lat weights, also removed deprecated code by @dkimpara in #9
  • Updating few bugs in model classes. Using units now in predict.py by @jsschreck in #11
  • zarrify.py script update by @sethmcg in #4
  • wrote PSDLoss, benchmarked values, added option fields to configs by @dkimpara in #14
  • Some quick improvments of draw_forecast within predict.py by @yingkaisha in #16
  • using os.path.expandvars in train.py,predict.py to enable generic save locs by @dkimpara in #17
  • Update predict.py, visualization_tools.py, and config file options by @yingkaisha in #20
  • Updated FSDP checkpointing by @jsschreck in #21
  • polar and laplacian diffusion filter module class by @WillyChap in #23
  • predict.py revamp - viz config options, async image generation, xarray creation, laplacian filtering, CPU-only runtime compatible by @dkimpara in #27
  • Modifications of predict.py by @jsschreck in #28
  • predict.py - reworked xr save format, using logger by @dkimpara in #29
  • Fixing FSDP revert; adding multi-step trajectory trainer by @jsschreck in #31
  • Quantile static by @WillyChap in #32
  • Updates to xformer model class and xformer configurations by @jsschreck in #34
  • Model base class by @dkimpara in #35
  • FuXi bug fix + Colorbar adjustments for visualization_tools by @yingkaisha in #36
  • TOA var and Quantile scaling, by @WillyChap in #37
  • Predict.py now supports FSDP by @jsschreck in #38
  • Adding static inputs by @jsschreck in #39
  • Conus404 data loader by @sethmcg in #22
  • adding transform/inverse for bscaler data by @WillyChap in #40
  • Addition of CONUS404 data, model, and training scripts by @jsschreck in #41
  • Updated FSDP-related bugs by @jsschreck in #42
  • edge case fix by @WillyChap in #44
  • Graph residual transformer with sparse edge calculation by @djgagne in #26
  • KE and spectrum visualization diagnostics + data conversions by @dkimpara in #43
  • Adding the replay buffer code as example by @jsschreck in #45
  • Adding Climate run capabilites by @WillyChap in #46
  • Fuxi update by @jsschreck in #47
  • Start of major refactoring of code base by @jsschreck in #48
  • TOA data type conversion and input tensor shape comment by @kanz76 in #51
  • FuXi model updates and data pipeline initial work by @yingkaisha in #52
  • Added Swin transformer model + new rollout script by @jsschreck in #53
  • The new data pipeline by @yingkaisha in #55
  • Data pipeline update with minor bugfix and documentation by @yingkaisha in #58
  • NetCDF metadata, docs, environment fixes by @djgagne in #56
  • Conus404 transforms by @sethmcg in #54
  • add 6hourly cached option by @WillyChap in #57
  • Emergency bug fix for the new roll-out and metadata loader by @yingkaisha in #60
  • update yaml to work by @WillyChap in #61
  • Add 6 hourly run configs with the new data pipeline + minor bugfix by @yingkaisha in #62
  • rollout_to_netcdf_new.py that works with the new data pipeline by @yingkaisha in #64
  • Updates to multi-step training a la Brenowitz scheme by @jsschreck in #63
  • Bugfix on rollout_to_netcdf_new.py + adding example.yml by @yingkaisha in #67
  • Memory optimization for one_shot=True + minor corrections for the example.yml by @yingkaisha in #69
  • Initialization of credit.trainers by @jsschreck in #65
  • Updates to the multi-step training code by @jsschreck in #70
  • new lat weighting by @WillyChap in #66
  • Add dynamic_forcing_variables into the data workflow by @yingkaisha in #73
  • Initial code for the Graph Residual Transformer + GRU model by @kanz76 in #72
  • Added reload_epoch option to the trainers by @jsschreck in #76
  • minor adjustments on credit.model + update configs by @yingkaisha in #74
  • Bugfix on model checkpointing and epoch update by @yingkaisha in #77
  • New solar processing and bridgescaler state scaler by @djgagne in #78
  • Updating the swin transformer by @jsschreck in #80
  • Adding parser and input data checks for config.yml + some minor fixes by @yingkaisha in #79
  • Model check-pointing and early-stopping fixes for base_trainer by @yingkaisha in #81
  • Bug fixes and few updates with variable weights by @jsschreck in #82
  • Small bugfix on parsing one_shot=False by @yingkaisha in #88
  • Updates to multi-step training by @jsschreck in #87
  • Adding arXiv paper production run configs + documentation by @yingkaisha in #89
  • [In Progress]: PBS Script and some minor improvements. by @negin513 in #90
  • few typos by @negin513 in #92
  • Fixes to new MPI launch; linting; reorganizing distributed by @jsschreck in #93
  • add lev functionality by @WillyChap in #84
  • Adding NVIDIA-makani scheme for multi-step training / rollout script for computing metrics on forecasts by @jsschreck in #91
  • multistep configurations + optimized cached dataset workflow + old script clean-up by @yingkaisha in #94
  • Updated learning rate scheduler and metrics handling by @jsschreck in #95
  • Notebooks and new github actions workflow by @djgagne in #97
  • Updated README with credit-derecho installation instructions added by @jsschreck in #99
  • Updated MPI support rollout_metrics by @jsschreck in #100
  • Emergency fix on rollout_to_netcdf.py and rollout_metrics by @yingkaisha in #104
  • Bugfix on trainerERA5_multistep_grad_accum.py by @yingkaisha in #106
  • Infrastructure for post-processing blocks (e.g. SKEBS, mass correction, laplacian filtering) by @dkimpara in #105
  • fixing environment_[device].yml specs by @dkimpara in #108
  • Major credit.postblock update + major credit.physics_core update + major bugfix + minor code optimization and cleaning by @yingkaisha in #86
  • Bugfix on the use of diagnostic variables in trainerERA5_multistep_grad_accum.py by @yingkaisha in #111
  • Fix NCLL_NET_GDR_LEVEL environment variable in credit.pbs.py by @kanz76 in #115
  • PHB does not work. Revert to PBH by @kanz76 in #118
  • credit.postblock major updates on GlobalMassFixer and GlobalEnergyFixer by @yingkaisha in #110
  • update unet init and call in models/init.py by @dkimpara in #119
  • updating the periodic padding function to roll then reflect lats by @WillyChap in #117
  • edits to pbs.py to improve usability by @dkimpara in #114
  • URGENT bugfix: removed cwd keyword from subprocess.popen by @dkimpara in #124
  • Add Python unit testing for credit.models.Fuxi and credit.boundary_padding by @yingkaisha in #121
  • Pressure interpolation and a bunch of formatting changes by @djgagne in #125
  • removed cwd from popen again. by @dkimpara in #127
  • Bugfix on the production of diagnostic variables in rollout_to_netcdf.py + credit.postblock update + other minor bugfix by @yingkaisha in #120
  • Add torch.clamp to trainers + add drop_path in FuXi + bugfix FuXi when use_spectral_norm: False + minor updates on credit.parser by @yingkaisha in #130
  • Fix mpi in rollout_to_netcdf.py by @kanz76 in #132
  • Bugifx on random seed inconsistencies and keyword grad_max_norm by @yingkaisha in #133
  • Version changes by @djgagne in #136

New Contributors

Full Changelog: https://github.com/NCAR/miles-credit/commits/2024.1.0

Contributors

djgagne, yingkaisha, and 6 other contributors
Assets 2
Loading