download while on AWS creates one file

[itcarroll]

The earthaccess.download function appears to be (over)writing all data received to a single file, rather than to separate files in the provided directory, when working on AWS. This example uses earthaccess 0.9.0 on cryointhecloud.

import earthaccess

earthaccess.login()
results = earthaccess.search_data(short_name="PACE_OCI_L2_AOP_NRT", count=2)
earthaccess.download(results, "test")
!ls -l test

Granules found: 4148
 Getting 2 granules, approx download size: 0.07 GB
Accessing cloud dataset using dataset endpoint credentials: https://obdaac-tea.earthdatacloud.nasa.gov/s3credentials
Downloaded: test/PACE_OCI.20240409T203648.L2.OC_AOP.V1_0_0.NRT.nc
Downloaded: test/PACE_OCI.20240409T204148.L2.OC_AOP.V1_0_0.NRT.nc
-rw-r--r-- 1 jovyan jovyan 9516948 May  8 21:04 test

Instead of a "test" folder, I get one "test" file.

Expected behavior would be either to raise "STAHP! Wut R u DoINg?" or just download the files.

[chuckwondo]

@itcarroll, I suspect this is not specific to cryointhecloud, nor AWS in general.

You likely need to first create the target directory (I don't know if it will be created for you, which may be the problem). Delete the erroneous file, create the test directory, then try again.

If that still doesn't work, then in addition, specify "test/" instead of simply "test" in your call to download.

Please report back your findings. Whatever your findings, this appears to be behavior that we must correct (and likely clearly document as well).

[itcarroll]

Oh, I should have mentioned that the function works as documented, even creating the directory if it doesn't exist, without direct access.

[chuckwondo]

Oh, I should have mentioned that the function works as documented, even creating the directory if it doesn't exist, without direct access.

Oh, very interesting. Would you mind trying my suggestions above, just to see if/how things are affected in that environment, just so we have more information about how things work/don't work?

[mfisher87]

Oh, I should have mentioned that the function works as documented, even creating the directory if it doesn't exist, without direct access.

👀 Good catch!!!

[itcarroll]

Oh, I should have mentioned that the function works as documented, even creating the directory if it doesn't exist, without direct access.

Oh, very interesting. Would you mind trying my suggestions above, just to see if/how things are affected in that environment, just so we have more information about how things work/don't work?

Creating the folder first causes expected behavior (i.e. multiple files in the folder). Using "test/" as the argument is equivalent to "test".

It's possible, based on fsspec's get() documentation, that a fix could be changing store.py#L574 to

s3_fs.get(file, str(local_path) + "/")

although that's a bit flimsy looking.

[chuckwondo]

Oh, I should have mentioned that the function works as documented, even creating the directory if it doesn't exist, without direct access.

Oh, very interesting. Would you mind trying my suggestions above, just to see if/how things are affected in that environment, just so we have more information about how things work/don't work?

Creating the folder first causes expected behavior (i.e. multiple files in the folder). Using "test/" as the argument is equivalent to "test".

It's possible, based on fsspec's get() documentation, that a fix could be changing store.py#L574 to

s3_fs.get(file, str(local_path) + "/")

although that's a bit flimsy looking.

Cool. Would you mind trying one more test? Delete the files in the test dir and also remove the dir again, so it doesn't exist. Then try downloading with "test/" as the argument. I want to see if including the trailing slash will then cause fsspec to create the directory for you.

[itcarroll]

If a "test" folder exists, download works with "test" and "test/". If there is nothing called "test", one file called "test" is created for either argument.

Conversion to pathlib.Path and back to str seems like it eliminates the slash: str(pathlib.Path("test/")) is just test.

[chuckwondo]

After a bit of digging into fsspec, I see the problem, which boils down to this: either the local (destination) directory must exist (and be a directory) OR the specified path must include a trailing slash. Otherwise, the destination is interpreted as a file rather than a directory, thus resulting in the reported behavior in this issue.

[chuckwondo]

@itcarroll, I don't have a CryoCloud account (I just submitted a request), but I did attempt to reproduce the error in the MAAP environment.

For reference, I noticed this in your reported output:

 Getting 2 granules, approx download size: 0.07 GB
Accessing cloud dataset using dataset endpoint credentials: https://obdaac-tea.earthdatacloud.nasa.gov/s3credentials
Downloaded: test/PACE_OCI.20240409T203648.L2.OC_AOP.V1_0_0.NRT.nc
Downloaded: test/PACE_OCI.20240409T204148.L2.OC_AOP.V1_0_0.NRT.nc
-rw-r--r-- 1 jovyan jovyan 9516948 May  8 21:04 test

Here's the output that I got:

 Getting 2 granules, approx download size: 0.07 GB
QUEUEING TASKS | : 100%|█████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 4344.18it/s]
PROCESSING TASKS | : 100%|█████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:03<00:00,  1.97s/it]
COLLECTING RESULTS | : 100%|████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 59074.70it/s]
['test/PACE_OCI.20240409T203648.L2.OC_AOP.V1_0_0.NRT.nc', 'test/PACE_OCI.20240409T204148.L2.OC_AOP.V1_0_0.NRT.nc']

That resulted in the expected test dir being created, and the 2 files downloaded into that dir.

However, I noticed that my output was different than yours, and it turns out that when I ran the code, earthaccess did not correctly determine that I was "in region," thus did not use the S3 download URLs, and instead used the HTTPS URLs.

Therefore, I removed the test dir, and did this:

earthaccess.__store__.in_region = True

Then I ran the download again, and this time I reproduced the behavior you reported:

 Getting 2 granules, approx download size: 0.07 GB
Accessing cloud dataset using dataset endpoint credentials: https://obdaac-tea.earthdatacloud.nasa.gov/s3credentials
Downloaded: test/PACE_OCI.20240409T203648.L2.OC_AOP.V1_0_0.NRT.nc
Downloaded: test/PACE_OCI.20240409T204148.L2.OC_AOP.V1_0_0.NRT.nc
[PosixPath('test/PACE_OCI.20240409T203648.L2.OC_AOP.V1_0_0.NRT.nc'), PosixPath('test/PACE_OCI.20240409T204148.L2.OC_AOP.V1_0_0.NRT.nc')]

This resulted in a single file named test containing the contents of the 2nd file.

The difference is that when earthaccess thinks it's not running "in region," it takes a different path in the code, and calls the Store._download_onprem_granules method, and that method explicitly creates the local directory (test in this case) before downloading the files, whereas the "in region" code path does not create the directory beforehand.

[itcarroll]

That all checks out with my understanding, and I like the explicit fix. Thanks!

I don't know what the MAAP is, but perhaps you are also hitting #444.

[chuckwondo]

That all checks out with my understanding, and I like the explicit fix. Thanks!

I don't know what the MAAP is, but perhaps you are also hitting #444.

Sorry, FYI, here's MAAP: https://maap-project.org/

I suspect I'm indeed hitting #444.