You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
There is a significant performance problem that occurs when trying to convert a large grib file into a single COG. It seems like pyramiding data in GDAL takes exponential time (probably a bug).
Thus, there's an advantage to splitting up large grib files into multiple ImageCollections / COGS. At the same time, we'd like to group COGs that come from the same source together, so it's easy to access groups of variables as bands. For example, we don't want to write one COG per band, as that would make the access pattern more difficult.
The solution that @fredzyda, @j9sh264 and I have discussed for this is to make one COG / ImageCollection per layer of the grib file. As described here, cfgrib will naturally return multiple datasets when opening a mixed hypercubes of data in a single grib. These offer natural groupings to break up the expensive COG operation.
Let's implement this capability generically, where we default to normalized grib schemas (#184), and let users opt-in to creating multiple image collections in Earth Engine.
The text was updated successfully, but these errors were encountered:
There is a significant performance problem that occurs when trying to convert a large grib file into a single COG. It seems like pyramiding data in GDAL takes exponential time (probably a bug).
Thus, there's an advantage to splitting up large grib files into multiple ImageCollections / COGS. At the same time, we'd like to group COGs that come from the same source together, so it's easy to access groups of variables as bands. For example, we don't want to write one COG per band, as that would make the access pattern more difficult.
The solution that @fredzyda, @j9sh264 and I have discussed for this is to make one COG / ImageCollection per layer of the grib file. As described here,
cfgrib
will naturally return multiple datasets when opening a mixed hypercubes of data in a single grib. These offer natural groupings to break up the expensive COG operation.Let's implement this capability generically, where we default to normalized grib schemas (#184), and let users opt-in to creating multiple image collections in Earth Engine.
The text was updated successfully, but these errors were encountered: