Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add optional feature to group common hypercubes into image collections when loading grib2 data into Earth Engine. #264

Closed
alxmrs opened this issue Dec 2, 2022 · 0 comments · Fixed by #304
Assignees

Comments

@alxmrs
Copy link
Collaborator

alxmrs commented Dec 2, 2022

There is a significant performance problem that occurs when trying to convert a large grib file into a single COG. It seems like pyramiding data in GDAL takes exponential time (probably a bug).

Thus, there's an advantage to splitting up large grib files into multiple ImageCollections / COGS. At the same time, we'd like to group COGs that come from the same source together, so it's easy to access groups of variables as bands. For example, we don't want to write one COG per band, as that would make the access pattern more difficult.

The solution that @fredzyda, @j9sh264 and I have discussed for this is to make one COG / ImageCollection per layer of the grib file. As described here, cfgrib will naturally return multiple datasets when opening a mixed hypercubes of data in a single grib. These offer natural groupings to break up the expensive COG operation.

Let's implement this capability generically, where we default to normalized grib schemas (#184), and let users opt-in to creating multiple image collections in Earth Engine.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants