Skip to content

Commit

Permalink
Merge remote-tracking branch 'origin/main' into feature_rank_one_utils
Browse files Browse the repository at this point in the history
  • Loading branch information
sadamov committed Jun 6, 2024
2 parents 7f99788 + c8d3553 commit 334ab42
Show file tree
Hide file tree
Showing 18 changed files with 799 additions and 124 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/pre-commit.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
name: lint
name: Linting

on:
# trigger on pushes to any branch, but not main
Expand Down
45 changes: 45 additions & 0 deletions .github/workflows/run_tests.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
name: Unit Tests

on:
# trigger on pushes to any branch, but not main
push:
branches-ignore:
- main
# and also on PRs to main
pull_request:
branches:
- main

jobs:
build:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.9", "3.10", "3.11", "3.12"]

steps:
- uses: actions/checkout@v3
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
run: |
python -m pip install --upgrade pip
if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
pip install torch-geometric>=2.5.2
- name: Load cache data
uses: actions/cache/restore@v4
with:
path: data
key: ${{ runner.os }}-meps-reduced-example-data-v0.1.0
restore-keys: |
${{ runner.os }}-meps-reduced-example-data-v0.1.0
- name: Test with pytest
run: |
pytest -v -s
- name: Save cache data
uses: actions/cache/save@v4
with:
path: data
key: ${{ runner.os }}-meps-reduced-example-data-v0.1.0
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,14 @@
wandb
slurm_log*
saved_models
lightning_logs
data
graphs
*.sif
sweeps
test_*.sh
.vscode
*slurm*

### Python ###
# Byte-compiled / optimized / DLL files
Expand Down
15 changes: 15 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [unreleased](https://github.com/joeloskarsson/neural-lam/compare/v0.1.0...HEAD)

### Added
- Added tests for loading dataset, creating graph, and training model based on reduced MEPS dataset stored on AWS S3, along with automatic running of tests on push/PR to GitHub. Added caching of test data tp speed up running tests.
[/#38](https://github.com/mllam/neural-lam/pull/38)
@SimonKamuk

- Replaced `constants.py` with `data_config.yaml` for data configuration management
[\#31](https://github.com/joeloskarsson/neural-lam/pull/31)
Expand All @@ -27,6 +30,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Changed

- Robust restoration of optimizer and scheduler using `ckpt_path`
[\#17](https://github.com/mllam/neural-lam/pull/17)
@sadamov

- Updated scripts and modules to use `data_config.yaml` instead of `constants.py`
[\#31](https://github.com/joeloskarsson/neural-lam/pull/31)
@sadamov
Expand Down Expand Up @@ -65,6 +72,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
[\#29](https://github.com/mllam/neural-lam/pull/29)
@leifdenby

- change copyright formulation in license to encompass all contributors
[\#47](https://github.com/mllam/neural-lam/pull/47)
@joeloskarsson

- Fix incorrect ordering of x- and y-dimensions in comments describing tensor
shapes for MEPS data
[\#52](https://github.com/mllam/neural-lam/pull/52)
@joeloskarsson

## [v0.1.0](https://github.com/joeloskarsson/neural-lam/releases/tag/v0.1.0)

Expand Down
2 changes: 1 addition & 1 deletion LICENSE.txt
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
MIT License

Copyright (c) 2023 Joel Oskarsson, Tomas Landelius
Copyright (c) 2023 Neural-LAM Contributors

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
Expand Down
5 changes: 5 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
![Linting](https://github.com/mllam/neural-lam/actions/workflows/pre-commit.yml/badge.svg)
![Automatic tests](https://github.com/mllam/neural-lam/actions/workflows/run_tests.yml/badge.svg)

<p align="middle">
<img src="figures/neural_lam_header.png" width="700">
</p>
Expand Down Expand Up @@ -279,6 +282,8 @@ pre-commit run --all-files
```
from the root directory of the repository.

Furthermore, all tests in the ```tests``` directory will be run upon pushing changes by a github action. Failure in any of the tests will also reject the push/PR.

# Contact
If you are interested in machine learning models for LAM, have questions about our implementation or ideas for extending it, feel free to get in touch.
You can open a github issue on this page, or (if more suitable) send an email to [joel.oskarsson@liu.se](mailto:joel.oskarsson@liu.se).
6 changes: 3 additions & 3 deletions create_grid_features.py
Original file line number Diff line number Diff line change
Expand Up @@ -29,14 +29,14 @@ def main():
# -- Static grid node features --
grid_xy = torch.tensor(
np.load(os.path.join(static_dir_path, "nwp_xy.npy"))
) # (2, N_x, N_y)
) # (2, N_y, N_x)
grid_xy = grid_xy.flatten(1, 2).T # (N_grid, 2)
pos_max = torch.max(torch.abs(grid_xy))
grid_xy = grid_xy / pos_max # Divide by maximum coordinate

geopotential = torch.tensor(
np.load(os.path.join(static_dir_path, "surface_geopotential.npy"))
) # (N_x, N_y)
) # (N_y, N_x)
geopotential = geopotential.flatten(0, 1).unsqueeze(1) # (N_grid,1)
gp_min = torch.min(geopotential)
gp_max = torch.max(geopotential)
Expand All @@ -46,7 +46,7 @@ def main():
grid_border_mask = torch.tensor(
np.load(os.path.join(static_dir_path, "border_mask.npy")),
dtype=torch.int64,
) # (N_x, N_y)
) # (N_y, N_x)
grid_border_mask = (
grid_border_mask.flatten(0, 1).to(torch.float).unsqueeze(1)
) # (N_grid, 1)
Expand Down
4 changes: 2 additions & 2 deletions create_mesh.py
Original file line number Diff line number Diff line change
Expand Up @@ -153,7 +153,7 @@ def prepend_node_index(graph, new_index):
return networkx.relabel_nodes(graph, to_mapping, copy=True)


def main():
def main(input_args=None):
parser = ArgumentParser(description="Graph generation arguments")
parser.add_argument(
"--data_config",
Expand Down Expand Up @@ -186,7 +186,7 @@ def main():
default=0,
help="Generate hierarchical mesh graph (default: 0, no)",
)
args = parser.parse_args()
args = parser.parse_args(input_args)

# Load grid positions
config_loader = config.Config.from_file(args.data_config)
Expand Down
Loading

0 comments on commit 334ab42

Please sign in to comment.