How can I use S3 links when earthaccess does not detect that I'm in-region (in us-west-2)?

[chuckwondo]

Recently, via Slack, Shane Coffield (at GSFC) indicated that he ran into a situation where he used code similar to the following in an environment within the AWS us-west-2 region, but earthaccess was unable to determine this to be the case, and thus used HTTPS rather than S3 urls:

import earthaccess
import xarray as xr

earthaccess.login()

results = earthaccess.search_data(
    short_name='VNP03IMG',
    bounding_box=(-125, 31, -102, 49.2),
    temporal=('2020-06-01', '2020-06-02'),
    count=5
)
files = earthaccess.open(results, provider="LAADS")
print(files[0])
xr.open_dataset(files[0], engine='h5netcdf', group='geolocation_data')

This printed the following:

<File-like object HTTPSFileSystem, https://data.laadsdaac.earthdatacloud.nasa.gov/prod-lads/VNP03IMG/VNP03IMG.A2020153.0812.002.2021125170327.nc>

So the question is, knowing that we are indeed running in us-west-2, how can we get earthaccess to do the right thing by using S3 rather than HTTPS, since it's ability to detect whether or not we're "in region" is not particularly robust?

[chuckwondo]

A very simple way to get this to work is to add the following line after the call to earthaccess.login() (suggested by David Giles):

earthaccess.__store__.in_region = True

However, use the approach above at your own risk. I recommend against this approach because __store__ is not part of the library's "public" API. Thus, it is subject to change without notice, and there is no promise of any sort of backward compatibility, which is generally the case for the non-public API of any library.

Instead, I recommend the following approach:

import earthaccess
import xarray as xr

earthaccess.login()

results = earthaccess.search_data(
    short_name='VNP03IMG',
    bounding_box=(-125, 31, -102, 49.2),
    temporal=('2020-06-01', '2020-06-02'),
    count=5
)

# Note: as of eartaccess 0.11.0, use get_s3_filesystem instead of get_s3fs_session.
# The 2 functions behave identically, but the new name accurately reflects the return type.
fs = earthaccess.get_s3fs_session(provider="LAADS")
s3_urls = [s3_url for result in results for s3_url in result.data_links(access="direct")]
print(s3_urls[0])

with (
    fs.open(s3_urls[0]) as nc,
    xr.open_dataset(nc, engine='h5netcdf', group='geolocation_data') as ds
):
    # Do whatever you need to do with `ds` here.
    print(ds)

This prints the following S3 url (ignoring the dataset output):

s3://prod-lads/VNP03IMG/VNP03IMG.A2020153.0812.002.2021125170327.nc

In addition to not accessing a "private" object (earthaccess.__store__), it also makes use of a context manager (the with block) to ensure that the open resources are properly closed and released.

Unfortunately, using earthaccess.open is not conducive to using a context manager. Although it can be done, it's rather clumsy, and thus I suspect you won't see anybody using earthaccess.open along with a context manager.

The downside of this is most notable within a Jupyter notebook, particularly when opening many files, especially when fetching large portions of each file, as this will likely produce memory problems due to the unreleased resources.

The following, similar code, can be used for those who need to download files, rather than open them directly (via xarray, for example):

fs = earthaccess.get_s3fs_session(provider="LAADS")
s3_urls = [s3_url for result in results for s3_url in result.data_links(access="direct")]
filenames = [s3_url.split("/")[-1] for s3_url in s3_urls]

for s3_url, filename in zip(s3_urls, filenames):
    fs.download(s3_url, filename)

[chuckwondo]

You don't load the data all at once. Do all of your data filtering within the context manager and xarray won't load any data until you explicitly tell it to load, or you call an xarray function that would trigger loading, but only the filtered/subsetted data would be loaded.

[chuckwondo]

If you can supply an example of what you currently do, perhaps I can suggest how to tweak it. Otherwise, feel free to simply not use a context manager. Things will still work. You just might run into a memory issue that careful use of context managers might help you avoid.

[itcarroll]

Extending your example:

with (
    fs.open(s3_urls[0]) as nc,
    xr.open_dataset(nc, engine='h5netcdf', group='geolocation_data', chunks={}) as ds
):
    da = ds["land_water_mask"].sel({"number_of_lines": 0})

This is not sufficient to do anything with da outside the context manager. Subsequently calling da.mean().compute(), for instance, raises ValueError: I[/O](https://oss.oel.smce.nasa.gov/O) operation on closed file..

Can you use the context manager without needing to call compute (or load if not using dask) from within it? That's what I meat when I said it looks like you have to load all the data at once. I want da to be backed by a dask.array and not a numpy.array, but don't see how I could do that with the context manager.

I understand its not essential to use a context manager, but since there seem to be benefits I would if it's compatible with lazy loading.

I also realize I'm digressing from your core recommendation on not using __store__, which is good suggestion!

[chuckwondo]

Your digression is a welcome discussion, as it is not necessarily obvious as to how to proceed if you do indeed want to make use of context managers.

In your example, your da value is still in an "uncomputed" state when the context manager exits, so what you are experiencing in your attempt to then use da after exiting the context manager is to be expected (because no data has yet been read before exiting the context manager), so this is actually a perfect example to help clarify things.

What you need to do is perform some sort of computation that forces/requires xarray to load data. That's exactly what the call to compute will do (and, of course, a direct call to load will also force data to be read), so you would need to call da.mean().compute() within the context manager to trigger reading the data before the file is closed implicitly by exiting the context manager.

Of course, if there are many things you want to "compute" from da, then you would either have to do all such computations within your context manager, or alternatively use a new context manager at each point you wish to perform another computation, if it would be too awkward to jam all computations within the same context manager.

[chuckwondo]

I see that some of my earlier wording was poorly stated:

You can use it in other cells, but within the context manager you should select the data you need so that xarray pulls in what you need prior to exiting the context manager, then you can use ds elsewhere.

That requires clarification. Specifically, since exiting the context manager closes the file, you cannot use ds elsewhere unless you force data loading/reading (load). However, if you have not applied enough filtering/selecting to ds to whittle down how much data would actually be loaded, this might not be the best idea, as you might simply load too much data. Instead, you likely want to perform some sort of computation with your whittled data prior to exiting the context manager, and then use your computed value outside of the context manager.

Of course, rather than using a context manager, you could explicitly call ds.close() explicitly at a point in your code where you're done using ds, but the downside of that approach is that you're either too likely to forget to do that, or you do remember, but then rearrange your code and fail to also move your call to ds.close() to the right spot. Further, if your call to ds.close() is in a cell that you never end up running, it does you no good. This is precisely why using a context manager is better for aiding resource management.

Again, this may not be necessary at all, but if you do end up having memory/resource problems with your notebooks, careful use of context managers may help mitigate the problem, but may not be the only thing you need to do to reduce memory/resource consumption.