Use Range Requests for Flatgeobuf to retrieve a subset of a given dataset

[brianbancroft]

I enjoy the simplicity and consistency of the loaders.gl library, but one of the most useful features of the flatgeobuf format is the ability to retrieve subsets of the source data by geographic bounding box, which the library accomplishes through HTTP range requests. I have looked through the codebase and documentation, and have not found any similar ability in loaders.gl.

The following is an example of how flatgeobuf uses geographic bounding boxes to load data from a gigabytes-sized source:
https://github.com/flatgeobuf/flatgeobuf/blob/master/examples/maplibre/large.html#L83

As far as my own use-cases, there are times when I reach for a geojson-type source over other sources when it comes to displaying on a mapbox/maplibre/ol map, esp when it comes to maintaining large sets of data, or when I want the labels to show on a map at all times. Flatgeobuf has filled that need, for the clunkiness of its node/js implementation. If loaders.gl/flatgeobuf allows for requests by bbox, I believe that it would make your already-clean implementation much more popular for this niche.

[kylebarron]

You're correct that we don't today support this. I think the ideal way for this to work is with a multi-step API so that an initial loader call can read just the flatgeobuf header. Then the user can inspect the returned metadata to see whether the flatgeobuf file supports geospatial filters (i.e. only if the flatgeobuf file contains a spatial index), as well as the full bounding box of the file.

We could also support this in our existing "single-shot" API, though if the file didn't contain the required metadata we might end up reading the whole file anyways

[brianbancroft]

Given an assumption of "the user expects that this file is streamable and is produced in a manner which allows streaming", would it make more sense to provide an error in the event that the required metadata is missing in that part of the file?

I can't speak for my expertise on this topic, but I've been experimenting with the files in the same way a kid experiments with lego blocks, and I've never had an issue with not having that metadata when I create a file. So far, I've only used GDAL or QGIS (which uses GDAL) though, so I have big selection bias here.

[kylebarron]

I've never had an issue with not having that metadata when I create a file

The flatgeobuf spec can but does not always have a spatial index. (e.g. in GDAL it's a layer creation option) The spatial index is not required for streaming; it's only required for random reads from different parts of the file.

[brianbancroft]

Gotcha