New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Added a flag to allow for level-wise ingestion of data. #304

Merged

j9sh264 merged 24 commits into main from mv-group-common-hypercubes

Dec 7, 2023

Collaborator

j9sh264 commented Mar 17, 2023

In this PR I have introduced a new flag named --group_common_hypercubes which allows to ingest data in a level-wise format into the Earth engine.

For more context refer to this issue.

j9sh264 added 4 commits

December 7, 2022 07:29


          Initial commit - COG naming as {filename}_level_{level_no}

63d9b94


          changed the naming convention to {asset_name}_level_{level}_{height}

e0b5161


          Code cleanup, removed the repeated code blocks.

71e94d6

Added checks for BigQuery.


          Addressed internal team review comments.

0a9af55

j9sh264 commented

View reviewed changes

weather_mv/loader_pipeline/ee.py Show resolved Hide resolved

j9sh264 requested review from alxmrs, mahrsee1997 and deepgabani8

March 17, 2023 06:47

j9sh264 added 2 commits

March 20, 2023 08:00


          Merge 'main' into mv-group-common-hypercubes

fda5c96


          flake8 changes

8466b99

alxmrs reviewed

View reviewed changes

Collaborator

alxmrs left a comment

A thought on the design of this change.

weather_mv/loader_pipeline/sinks.py Outdated Show resolved Hide resolved

alxmrs requested a review from fredzyda

March 28, 2023 18:38

fredzyda reviewed

View reviewed changes

Collaborator

fredzyda left a comment

I agree with Alex.

weather_mv/loader_pipeline/sinks.py Outdated Show resolved Hide resolved

j9sh264 added 5 commits

May 17, 2023 06:32


          New changes to maintain backward compatibility

87e9965


          BigQuery changes

b2942f7


          Merge branch 'main' into mv-group-common-hypercubes

acf5395


          Linting changes

8c8da57


          bigquery minor yield change

a8ab4e8

mahrsee1997 requested a review from alxmrs

May 18, 2023 16:51

alxmrs approved these changes

View reviewed changes

Collaborator

alxmrs left a comment

Let's fix a few small issues first; otherwise I think this is good to go :)

weather_mv/loader_pipeline/bq.py Outdated Show resolved Hide resolved

weather_mv/loader_pipeline/ee.py Outdated Show resolved Hide resolved

weather_mv/loader_pipeline/ee.py Show resolved Hide resolved

weather_mv/loader_pipeline/sinks.py Outdated Show resolved Hide resolved

weather_mv/loader_pipeline/sinks.py Outdated Show resolved Hide resolved

mahrsee1997 requested a review from fredzyda

May 18, 2023 16:59

j9sh264 added 5 commits

May 22, 2023 06:10


          Added test cases

ebdb0e3


          Addressed comments

8ad45a7


          Filename correction

3b5763d


          Merge branch 'main' into mv-group-common-hypercubes

ef7a348


          Removed zarr argument check from bq.py

18e738d

mahrsee1997 requested changes

View reviewed changes

weather_mv/test_data/test_data_grib_multi_levels.grib2 Outdated Show resolved Hide resolved

weather_mv/loader_pipeline/sinks.py Outdated Show resolved Hide resolved

weather_mv/loader_pipeline/ee.py Show resolved Hide resolved

weather_mv/loader_pipeline/sinks.py Outdated Show resolved Hide resolved

weather_mv/loader_pipeline/sinks.py Outdated Show resolved Hide resolved


          Addressed Rahul's comments

deepgabani8 reviewed

View reviewed changes

weather_mv/loader_pipeline/sinks.py Outdated Show resolved Hide resolved

deepgabani8 reviewed

View reviewed changes

weather_mv/loader_pipeline/sinks.py Outdated Show resolved Hide resolved

deepgabani8 reviewed

View reviewed changes

weather_mv/loader_pipeline/sinks.py Outdated Show resolved Hide resolved

j9sh264 added 2 commits

June 13, 2023 12:08


          Resolved comments

1a63361


          Version bump change

8780a79

j9sh264 linked an issue

that may be closed by this pull request

Add optional feature to group common hypercubes into image collections when loading grib2 data into Earth Engine. #264

Closed


          Merge branch 'main' into mv-group-common-hypercubes

437fc58

Piyush-Ingale mentioned this pull request

added support for source data with multiple time dimensions #340

Open

j9sh264 requested a review from mahrsee1997

July 12, 2023 14:31

j9sh264 added 4 commits

October 17, 2023 10:12


          Merge branch 'main' into mv-group-common-hypercubes

943cae0


          flake8 changes

6abe689


          Minor change to solve sinks test cases.

a3e54d7


          Updated position of group_common_hypercubes function arg.

464a20d

deepgabani8 approved these changes

View reviewed changes

j9sh264 removed the request for review from fredzyda

December 2, 2023 01:10

mahrsee1997 approved these changes

View reviewed changes

Collaborator

mahrsee1997 left a comment

LGTM.

j9sh264 merged commit 4e4ad1f into main

7 checks passed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet