Potentially (M)RE for 'hanging' during rechunk based on CMIP6 recipes. #715
Comments
|
Ok I got an example running here (successful submission log here). This is running on LEAP dataflow, but might be not to hard to run locally? I was a bit hesitant for now since it will cache about 100GB of data. I can probably from there test/probe a bit more to reduce the complexity further. |
|
Ok it has been running for about an hour, and has scaled up to ~6, then back to 3 workers, but they all seem to idle around after a while 
FWIW the console indicates that this is stuck on the rechunk stage But from experience I know that these graphical representations can often be misleading. |
|
I think this is a good candidate! Will report back after a few hours. |
|
Some strategies to further simplify this:
|
|
In the other hanging recipe, it seemed like about three months of daily data would result in hanging. |
|
Ill see tomorrow if there is an upper bound to it not hanging in this case, but I am not convinced this is representative of a 'stall'. Jobs been running for 4 hours and basically doing nothing AFAICT: |
|
If (and thats a big if, see above) we can believe the graph diagram, it is actually stuck on the group by key stage 🤔 
|
jbusecke
changed the title
|
I am moving the example to a separate repo to keep things more neat. |
|
Also I just cancelled the job from yesterday after it running for 19 hours without really doing anything... |
|
Currently trying out some variations of my failcase here. Will report back on what I find. |
|
Ok this is interesting. It seems that even only using 3 files (of 33) is enough to trigger this! If that holds true this would make this example a lot more minimal. |
|
Got 4 examples in the oven I ll have to wait ~1 hour to be sure things are stuck. |
|
Ok it seems that the complex rechunking structure has something to do with the problem (as I previously suspected), when running the recipe with chunks along the time axis only both recipes succeeded! This also gives us an approximate time that things "should take"? So from now on I will assume that any recipe with runtime of many hours is actually stuck. |
|
I am also calling that the short example is in fact stuck! |
|
Next I will try to run the |
|
I went into a different direction and I think I am now able to reproduce the problem without my dynamical rechunking. Simply setting the chunking 'orthogonal' (all chunks along lon, while concatting along time) does reproduce the behavior with only 2! files. |
|
Notes on cases to test:
|
|
@norlandrhagen and I are in the middle of a pair coding session. I will jot down some notes (and edit later if needed). We are currently trying to run two recipes locally
|
|
Ok I just ran both examples with the direct runner via pgf-runner (on the LEAP Jupyterhub with 16 cores) like this (Thx to @ranchodeluxe for the instructions):
These both worked! So I am not sure if my problem is in fact the same problem that @norlandrhagen and @ranchodeluxe encountered in #710. I wonder if somebody can run this on flink to add another datapoint to our fail/runner matrix? |
|
Ok so with input from @cisaacstern I tried the same thing except I used a gcs bucket for cache and output which also did not fail |
|
Ok @norlandrhagen and I just wrapped up. We devised a few new recipe_ids to test some other things:
"Workflow failed. Causes: S05:Creating CMIP6.CMIP.CMCC.CMCC-ESM2.historical.r1i1p1f1.3hr.pr.gn.v20210114|OpenURLWithFSSpec|OpenWithXarray|Preprocessor|StoreToZarr|ConsolidateDimensionCoordinates|ConsolidateMetadata/Creating CMIP6.CMIP.CMCC.CMCC-ESM2.historical.r1i1p1f1.3hr.pr.gn.v20210114/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Read+Creating CMIP6.CMIP.CMCC.CMCC-ESM2.historical.r1i1p1f1.3hr.pr.gn.v20210114|OpenURLWithFSSpec|OpenWithXarray|Preprocessor|StoreToZarr|ConsolidateDimensionCoordinates|ConsolidateMetadata/Creating CMIP6.CMIP.CMCC.CMCC-ESM2.historical.r1i1p1f1.3hr.pr.gn.v20210114/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/GroupByWindow+Creating CMIP6.CMIP.CMCC.CMCC-ESM2.historical.r1i1p1f1.3hr.pr.gn.v20210114|OpenURLWithFSSpec|OpenWithXarray|Preprocessor|StoreToZarr|ConsolidateDimensionCoordinates|ConsolidateMetadata/Creating CMIP6.CMIP.CMCC.CMCC-ESM2.historical.r1i1p1f1.3hr.pr.gn.v20210114/MaybeReshuffle/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps)+Creating CMIP6.CMIP.CMCC.CMCC-ESM2.historical.r1i1p1f1.3hr.pr.gn.v20210114|OpenURLWithFSSpec|OpenWithXarray|Preprocessor|StoreToZarr|ConsolidateDimensionCoordinates|ConsolidateMetadata/Creating CMIP6.CMIP.CMCC.CMCC-ESM2.historical.r1i1p1f1.3hr.pr.gn.v20210114/MaybeReshuffle/Reshuffle/RemoveRandomKeys+Creating CMIP6.CMIP.CMCC.CMCC-ESM2.historical.r1i1p1f1.3hr.pr.gn.v20210114|OpenURLWithFSSpec|OpenWithXarray|Preprocessor|StoreToZarr|ConsolidateDimensionCoordinates|ConsolidateMetadata/Creating CMIP6.CMIP.CMCC.CMCC-ESM2.historical.r1i1p1f1.3hr.pr.gn.v20210114/Map(decode)+Creating CMIP6.CMIP.CMCC.CMCC-ESM2.historical.r1i1p1f1.3hr.pr.gn.v20210114|OpenURLWithFSSpec|OpenWithXarray|Preprocessor|StoreToZarr|ConsolidateDimensionCoordinates|ConsolidateMetadata/OpenURLWithFSSpec/MapWithConcurrencyLimit/open_url+Creating CMIP6.CMIP.CMCC.CMCC-ESM2.historical.r1i1p1f1.3hr.pr.gn.v20210114|OpenURLWithFSSpec|OpenWithXarray|Preprocessor|StoreToZarr|ConsolidateDimensionCoordinates|ConsolidateMetadata/OpenWithXarray/Open with Xarray+Creating CMIP6.CMIP.CMCC.CMCC-ESM2.historical.r1i1p1f1.3hr.pr.gn.v20210114|OpenURLWithFSSpec|OpenWithXarray|Preprocessor|StoreToZarr|ConsolidateDimensionCoordinates|ConsolidateMetadata/Preprocessor/Fix coordinates+Creating CMIP6.CMIP.CMCC.CMCC-ESM2.historical.r1i1p1f1.3hr.pr.gn.v20210114|OpenURLWithFSSpec|OpenWithXarray|Preprocessor|StoreToZarr|ConsolidateDimensionCoordinates|ConsolidateMetadata/Preprocessor/Sanitize Attrs+Creating CMIP6.CMIP.CMCC.CMCC-ESM2.historical.r1i1p1f1.3hr.pr.gn.v20210114|OpenURLWithFSSpec|OpenWithXarray|Preprocessor|StoreToZarr|ConsolidateDimensionCoordinates|ConsolidateMetadata/StoreToZarr/DetermineSchema/MapTuple()+Creating CMIP6.CMIP.CMCC.CMCC-ESM2.historical.r1i1p1f1.3hr.pr.gn.v20210114|OpenURLWithFSSpec|OpenWithXarray|Preprocessor|StoreToZarr|ConsolidateDimensionCoordinates|ConsolidateMetadata/StoreToZarr/DetermineSchema/CombineGlobally(CombineXarraySchemas)/KeyWithVoid+Creating CMIP6.CMIP.CMCC.CMCC-ESM2.historical.r1i1p1f1.3hr.pr.gn.v20210114|OpenURLWithFSSpec|OpenWithXarray|Preprocessor|StoreToZarr|ConsolidateDimensionCoordinates|ConsolidateMetadata/StoreToZarr/DetermineSchema/CombineGlobally(CombineXarraySchemas)/CombinePerKey/GroupByKey+Creating CMIP6.CMIP.CMCC.CMCC-ESM2.historical.r1i1p1f1.3hr.pr.gn.v20210114|OpenURLWithFSSpec|OpenWithXarray|Preprocessor|StoreToZarr|ConsolidateDimensionCoordinates|ConsolidateMetadata/StoreToZarr/DetermineSchema/CombineGlobally(CombineXarraySchemas)/CombinePerKey/Combine/Partial+Creating CMIP6.CMIP.CMCC.CMCC-ESM2.historical.r1i1p1f1.3hr.pr.gn.v20210114|OpenURLWithFSSpec|OpenWithXarray|Preprocessor|StoreToZarr|ConsolidateDimensionCoordinates|ConsolidateMetadata/StoreToZarr/DetermineSchema/CombineGlobally(CombineXarraySchemas)/CombinePerKey/GroupByKey/Write failed., The job failed because a work item has failed 4 times. Look in previous log entries for the cause of each one of the 4 failures. If the logs only contain generic timeout errors related to accessing external resources, such as MongoDB, verify that the worker service account has permission to access the resource's subnetwork. For more information, see https://cloud.google.com/dataflow/docs/guides/common-errors. The work item was attempted on these workers:
Ill pick this up next week, but any feedback is very welcome. |
|
From the worker logs of the OG |
Given this beautiful traceback I say the next steps to reproduce are:
|
|
@moradology found this: fsspec/gcsfs#379 |
|
I just confirmed that the problem is not resolved when only running on one worker. @moradology and I just stumbled upon some output that might a bug in the groupby logic of rechunk. We will continue on this later and report back. |
|
Ok I think @moradology and I have honed in on the problem earlier today, and I was able to produce what I think is unintended behavior from the rechunking stage locally. How and why that would only stall on dataflow still eludes me, but I think regardless this is actually a pretty significant bug that needs to be dealt with before I can move on. Ok let me back up. So my main hypothesis is that the combining of fragments(subfragments?) is not actually properly working! I have evidence from the above example but I think it is more instructive to look at this new tiny example that shares the main attribute with the CMIP test examples: Target chunks that are orthogonal to the concat dim, requiring a complete rechunk. Take these two example datasets: import xarray as xr
a = xr.DataArray(
[[0,1],[2,3]],
dims=['x', 'time'],
coords={'x':[0,1], 'time':[0,1]}
).to_dataset(name='var').chunk({'time':1})
b = xr.DataArray(
[[4,5],[6,7]],
dims=['x', 'time'],
coords={'x':[0,1], 'time':[2,3]}
).to_dataset(name='var').chunk({'time':1})I am saving both of those files locally as netcdfs and 'simulate' the desired output zarr dataset: 
I then set up some boilerplate feedstock (I can make another repo if folks want to run this locally): with the following files:
and I also have a
I then run the following in the terminal The recipe runs with the following output (This is ugly AF and only for completeness. Ill fish out the important parts below):
Collecting pangeo-forge-esgf==0.1.1 (from -r feedstock/requirements.txt (line 1))
Using cached pangeo_forge_esgf-0.1.1-py3-none-any.whl.metadata (4.2 kB)
Collecting pangeo-forge-recipes==0.10.6 (from -r feedstock/requirements.txt (line 2))
Using cached pangeo_forge_recipes-0.10.6-py3-none-any.whl.metadata (4.5 kB)
Collecting dynamic-chunks==0.0.2 (from -r feedstock/requirements.txt (line 3))
Using cached dynamic_chunks-0.0.2-py3-none-any.whl.metadata (1.9 kB)
Collecting gcsfs (from -r feedstock/requirements.txt (line 4))
Using cached gcsfs-2024.3.1-py2.py3-none-any.whl.metadata (1.6 kB)
Collecting apache-beam[gcp] (from -r feedstock/requirements.txt (line 5))
Using cached apache_beam-2.55.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.6 kB)
Collecting aiohttp (from pangeo-forge-esgf==0.1.1->-r feedstock/requirements.txt (line 1))
Using cached aiohttp-3.9.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (7.4 kB)
Collecting requests (from pangeo-forge-esgf==0.1.1->-r feedstock/requirements.txt (line 1))
Using cached requests-2.31.0-py3-none-any.whl.metadata (4.6 kB)
Collecting tqdm (from pangeo-forge-esgf==0.1.1->-r feedstock/requirements.txt (line 1))
Using cached tqdm-4.66.2-py3-none-any.whl.metadata (57 kB)
Collecting backoff (from pangeo-forge-esgf==0.1.1->-r feedstock/requirements.txt (line 1))
Using cached backoff-2.2.1-py3-none-any.whl.metadata (14 kB)
Collecting cftime (from pangeo-forge-recipes==0.10.6->-r feedstock/requirements.txt (line 2))
Using cached cftime-1.6.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (8.6 kB)
Collecting dask>=2021.11.2 (from pangeo-forge-recipes==0.10.6->-r feedstock/requirements.txt (line 2))
Using cached dask-2024.3.1-py3-none-any.whl.metadata (3.8 kB)
Collecting fastparquet (from pangeo-forge-recipes==0.10.6->-r feedstock/requirements.txt (line 2))
Using cached fastparquet-2024.2.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.1 kB)
Collecting fsspec>=2023.4.0 (from fsspec[http]>=2023.4.0->pangeo-forge-recipes==0.10.6->-r feedstock/requirements.txt (line 2))
Using cached fsspec-2024.3.1-py3-none-any.whl.metadata (6.8 kB)
Collecting h5netcdf (from pangeo-forge-recipes==0.10.6->-r feedstock/requirements.txt (line 2))
Using cached h5netcdf-1.3.0-py3-none-any.whl.metadata (13 kB)
Collecting h5py>=3.3.0 (from pangeo-forge-recipes==0.10.6->-r feedstock/requirements.txt (line 2))
Using cached h5py-3.10.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (2.5 kB)
Collecting kerchunk>=0.0.7 (from pangeo-forge-recipes==0.10.6->-r feedstock/requirements.txt (line 2))
Using cached kerchunk-0.2.4-py3-none-any.whl.metadata (4.3 kB)
Collecting netcdf4 (from pangeo-forge-recipes==0.10.6->-r feedstock/requirements.txt (line 2))
Using cached netCDF4-1.6.5-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.8 kB)
Collecting numcodecs>=0.9.0 (from pangeo-forge-recipes==0.10.6->-r feedstock/requirements.txt (line 2))
Using cached numcodecs-0.12.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (2.8 kB)
Collecting xarray>=0.18.0 (from pangeo-forge-recipes==0.10.6->-r feedstock/requirements.txt (line 2))
Using cached xarray-2024.2.0-py3-none-any.whl.metadata (11 kB)
Collecting zarr>=2.12.0 (from pangeo-forge-recipes==0.10.6->-r feedstock/requirements.txt (line 2))
Using cached zarr-2.17.1-py3-none-any.whl.metadata (5.7 kB)
Collecting numpy (from dynamic-chunks==0.0.2->-r feedstock/requirements.txt (line 3))
Using cached numpy-1.26.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (61 kB)
Collecting decorator>4.1.2 (from gcsfs->-r feedstock/requirements.txt (line 4))
Using cached decorator-5.1.1-py3-none-any.whl.metadata (4.0 kB)
Collecting google-auth>=1.2 (from gcsfs->-r feedstock/requirements.txt (line 4))
Using cached google_auth-2.29.0-py2.py3-none-any.whl.metadata (4.7 kB)
Collecting google-auth-oauthlib (from gcsfs->-r feedstock/requirements.txt (line 4))
Using cached google_auth_oauthlib-1.2.0-py2.py3-none-any.whl.metadata (2.7 kB)
Collecting google-cloud-storage (from gcsfs->-r feedstock/requirements.txt (line 4))
Using cached google_cloud_storage-2.16.0-py2.py3-none-any.whl.metadata (6.1 kB)
Collecting crcmod<2.0,>=1.7 (from apache-beam[gcp]->-r feedstock/requirements.txt (line 5))
Using cached crcmod-1.7-py3-none-any.whl
Collecting orjson<4,>=3.9.7 (from apache-beam[gcp]->-r feedstock/requirements.txt (line 5))
Using cached orjson-3.9.15-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (49 kB)
Collecting dill<0.3.2,>=0.3.1.1 (from apache-beam[gcp]->-r feedstock/requirements.txt (line 5))
Using cached dill-0.3.1.1-py3-none-any.whl
Collecting cloudpickle~=2.2.1 (from apache-beam[gcp]->-r feedstock/requirements.txt (line 5))
Using cached cloudpickle-2.2.1-py3-none-any.whl.metadata (6.9 kB)
Collecting fastavro<2,>=0.23.6 (from apache-beam[gcp]->-r feedstock/requirements.txt (line 5))
Using cached fastavro-1.9.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.5 kB)
Collecting fasteners<1.0,>=0.3 (from apache-beam[gcp]->-r feedstock/requirements.txt (line 5))
Using cached fasteners-0.19-py3-none-any.whl.metadata (4.9 kB)
Collecting grpcio!=1.48.0,<2,>=1.33.1 (from apache-beam[gcp]->-r feedstock/requirements.txt (line 5))
Using cached grpcio-1.62.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.0 kB)
Collecting hdfs<3.0.0,>=2.1.0 (from apache-beam[gcp]->-r feedstock/requirements.txt (line 5))
Using cached hdfs-2.7.0-py3-none-any.whl.metadata (3.9 kB)
Collecting httplib2<0.23.0,>=0.8 (from apache-beam[gcp]->-r feedstock/requirements.txt (line 5))
Using cached httplib2-0.22.0-py3-none-any.whl.metadata (2.6 kB)
Collecting js2py<1,>=0.74 (from apache-beam[gcp]->-r feedstock/requirements.txt (line 5))
Using cached Js2Py-0.74-py3-none-any.whl.metadata (868 bytes)
Collecting jsonschema<5.0.0,>=4.0.0 (from apache-beam[gcp]->-r feedstock/requirements.txt (line 5))
Using cached jsonschema-4.21.1-py3-none-any.whl.metadata (7.8 kB)
Collecting jsonpickle<4.0.0,>=3.0.0 (from apache-beam[gcp]->-r feedstock/requirements.txt (line 5))
Using cached jsonpickle-3.0.3-py3-none-any.whl.metadata (7.3 kB)
Collecting objsize<0.8.0,>=0.6.1 (from apache-beam[gcp]->-r feedstock/requirements.txt (line 5))
Using cached objsize-0.7.0-py3-none-any.whl.metadata (12 kB)
Collecting packaging>=22.0 (from apache-beam[gcp]->-r feedstock/requirements.txt (line 5))
Using cached packaging-24.0-py3-none-any.whl.metadata (3.2 kB)
Collecting pymongo<5.0.0,>=3.8.0 (from apache-beam[gcp]->-r feedstock/requirements.txt (line 5))
Using cached pymongo-4.6.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (22 kB)
Collecting proto-plus<2,>=1.7.1 (from apache-beam[gcp]->-r feedstock/requirements.txt (line 5))
Using cached proto_plus-1.23.0-py3-none-any.whl.metadata (2.2 kB)
Collecting protobuf!=4.0.*,!=4.21.*,!=4.22.0,!=4.23.*,!=4.24.*,<4.26.0,>=3.20.3 (from apache-beam[gcp]->-r feedstock/requirements.txt (line 5))
Using cached protobuf-4.25.3-cp37-abi3-manylinux2014_x86_64.whl.metadata (541 bytes)
Collecting pydot<2,>=1.2.0 (from apache-beam[gcp]->-r feedstock/requirements.txt (line 5))
Using cached pydot-1.4.2-py2.py3-none-any.whl.metadata (8.0 kB)
Collecting python-dateutil<3,>=2.8.0 (from apache-beam[gcp]->-r feedstock/requirements.txt (line 5))
Using cached python_dateutil-2.9.0.post0-py2.py3-none-any.whl.metadata (8.4 kB)
Collecting pytz>=2018.3 (from apache-beam[gcp]->-r feedstock/requirements.txt (line 5))
Using cached pytz-2024.1-py2.py3-none-any.whl.metadata (22 kB)
Collecting regex>=2020.6.8 (from apache-beam[gcp]->-r feedstock/requirements.txt (line 5))
Using cached regex-2023.12.25-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (40 kB)
Collecting typing-extensions>=3.7.0 (from apache-beam[gcp]->-r feedstock/requirements.txt (line 5))
Using cached typing_extensions-4.10.0-py3-none-any.whl.metadata (3.0 kB)
Collecting zstandard<1,>=0.18.0 (from apache-beam[gcp]->-r feedstock/requirements.txt (line 5))
Using cached zstandard-0.22.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (2.9 kB)
Collecting pyarrow<15.0.0,>=3.0.0 (from apache-beam[gcp]->-r feedstock/requirements.txt (line 5))
Using cached pyarrow-14.0.2-cp311-cp311-manylinux_2_28_x86_64.whl.metadata (3.0 kB)
Collecting pyarrow-hotfix<1 (from apache-beam[gcp]->-r feedstock/requirements.txt (line 5))
Using cached pyarrow_hotfix-0.6-py3-none-any.whl.metadata (3.6 kB)
Collecting cachetools<6,>=3.1.0 (from apache-beam[gcp]->-r feedstock/requirements.txt (line 5))
Using cached cachetools-5.3.3-py3-none-any.whl.metadata (5.3 kB)
Collecting google-api-core<3,>=2.0.0 (from apache-beam[gcp]->-r feedstock/requirements.txt (line 5))
Using cached google_api_core-2.18.0-py3-none-any.whl.metadata (2.7 kB)
Collecting google-apitools<0.5.32,>=0.5.31 (from apache-beam[gcp]->-r feedstock/requirements.txt (line 5))
Using cached google_apitools-0.5.31-py3-none-any.whl
Collecting google-auth-httplib2<0.2.0,>=0.1.0 (from apache-beam[gcp]->-r feedstock/requirements.txt (line 5))
Using cached google_auth_httplib2-0.1.1-py2.py3-none-any.whl.metadata (2.1 kB)
Collecting google-cloud-datastore<3,>=2.0.0 (from apache-beam[gcp]->-r feedstock/requirements.txt (line 5))
Using cached google_cloud_datastore-2.19.0-py2.py3-none-any.whl.metadata (5.6 kB)
Collecting google-cloud-pubsub<3,>=2.1.0 (from apache-beam[gcp]->-r feedstock/requirements.txt (line 5))
Using cached google_cloud_pubsub-2.21.0-py2.py3-none-any.whl.metadata (9.3 kB)
Collecting google-cloud-pubsublite<2,>=1.2.0 (from apache-beam[gcp]->-r feedstock/requirements.txt (line 5))
Using cached google_cloud_pubsublite-1.9.0-py2.py3-none-any.whl.metadata (5.6 kB)
Collecting google-cloud-bigquery<4,>=2.0.0 (from apache-beam[gcp]->-r feedstock/requirements.txt (line 5))
Using cached google_cloud_bigquery-3.19.0-py2.py3-none-any.whl.metadata (8.9 kB)
Collecting google-cloud-bigquery-storage<3,>=2.6.3 (from apache-beam[gcp]->-r feedstock/requirements.txt (line 5))
Using cached google_cloud_bigquery_storage-2.24.0-py2.py3-none-any.whl.metadata (5.6 kB)
Collecting google-cloud-core<3,>=2.0.0 (from apache-beam[gcp]->-r feedstock/requirements.txt (line 5))
Using cached google_cloud_core-2.4.1-py2.py3-none-any.whl.metadata (2.7 kB)
Collecting google-cloud-bigtable<3,>=2.19.0 (from apache-beam[gcp]->-r feedstock/requirements.txt (line 5))
Using cached google_cloud_bigtable-2.23.0-py2.py3-none-any.whl.metadata (5.3 kB)
Collecting google-cloud-spanner<4,>=3.0.0 (from apache-beam[gcp]->-r feedstock/requirements.txt (line 5))
Using cached google_cloud_spanner-3.44.0-py2.py3-none-any.whl.metadata (10 kB)
Collecting google-cloud-dlp<4,>=3.0.0 (from apache-beam[gcp]->-r feedstock/requirements.txt (line 5))
Using cached google_cloud_dlp-3.16.0-py2.py3-none-any.whl.metadata (5.3 kB)
Collecting google-cloud-language<3,>=2.0 (from apache-beam[gcp]->-r feedstock/requirements.txt (line 5))
Using cached google_cloud_language-2.13.3-py2.py3-none-any.whl.metadata (5.3 kB)
Collecting google-cloud-videointelligence<3,>=2.0 (from apache-beam[gcp]->-r feedstock/requirements.txt (line 5))
Using cached google_cloud_videointelligence-2.13.3-py2.py3-none-any.whl.metadata (5.7 kB)
Collecting google-cloud-vision<4,>=2 (from apache-beam[gcp]->-r feedstock/requirements.txt (line 5))
Using cached google_cloud_vision-3.7.2-py2.py3-none-any.whl.metadata (5.2 kB)
Collecting google-cloud-recommendations-ai<0.11.0,>=0.1.0 (from apache-beam[gcp]->-r feedstock/requirements.txt (line 5))
Using cached google_cloud_recommendations_ai-0.10.10-py2.py3-none-any.whl.metadata (5.3 kB)
Collecting google-cloud-aiplatform<2.0,>=1.26.0 (from apache-beam[gcp]->-r feedstock/requirements.txt (line 5))
Using cached google_cloud_aiplatform-1.44.0-py2.py3-none-any.whl.metadata (27 kB)
Collecting aiosignal>=1.1.2 (from aiohttp->pangeo-forge-esgf==0.1.1->-r feedstock/requirements.txt (line 1))
Using cached aiosignal-1.3.1-py3-none-any.whl.metadata (4.0 kB)
Collecting attrs>=17.3.0 (from aiohttp->pangeo-forge-esgf==0.1.1->-r feedstock/requirements.txt (line 1))
Using cached attrs-23.2.0-py3-none-any.whl.metadata (9.5 kB)
Collecting frozenlist>=1.1.1 (from aiohttp->pangeo-forge-esgf==0.1.1->-r feedstock/requirements.txt (line 1))
Using cached frozenlist-1.4.1-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (12 kB)
Collecting multidict<7.0,>=4.5 (from aiohttp->pangeo-forge-esgf==0.1.1->-r feedstock/requirements.txt (line 1))
Using cached multidict-6.0.5-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.2 kB)
Collecting yarl<2.0,>=1.0 (from aiohttp->pangeo-forge-esgf==0.1.1->-r feedstock/requirements.txt (line 1))
Using cached yarl-1.9.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (31 kB)
Collecting click>=8.1 (from dask>=2021.11.2->pangeo-forge-recipes==0.10.6->-r feedstock/requirements.txt (line 2))
Using cached click-8.1.7-py3-none-any.whl.metadata (3.0 kB)
Collecting partd>=1.2.0 (from dask>=2021.11.2->pangeo-forge-recipes==0.10.6->-r feedstock/requirements.txt (line 2))
Using cached partd-1.4.1-py3-none-any.whl.metadata (4.6 kB)
Collecting pyyaml>=5.3.1 (from dask>=2021.11.2->pangeo-forge-recipes==0.10.6->-r feedstock/requirements.txt (line 2))
Using cached PyYAML-6.0.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (2.1 kB)
Collecting toolz>=0.10.0 (from dask>=2021.11.2->pangeo-forge-recipes==0.10.6->-r feedstock/requirements.txt (line 2))
Using cached toolz-0.12.1-py3-none-any.whl.metadata (5.1 kB)
Collecting importlib-metadata>=4.13.0 (from dask>=2021.11.2->pangeo-forge-recipes==0.10.6->-r feedstock/requirements.txt (line 2))
Using cached importlib_metadata-7.1.0-py3-none-any.whl.metadata (4.7 kB)
Collecting googleapis-common-protos<2.0.dev0,>=1.56.2 (from google-api-core<3,>=2.0.0->apache-beam[gcp]->-r feedstock/requirements.txt (line 5))
Using cached googleapis_common_protos-1.63.0-py2.py3-none-any.whl.metadata (1.5 kB)
Collecting oauth2client>=1.4.12 (from google-apitools<0.5.32,>=0.5.31->apache-beam[gcp]->-r feedstock/requirements.txt (line 5))
Using cached oauth2client-4.1.3-py2.py3-none-any.whl.metadata (1.2 kB)
Collecting six>=1.12.0 (from google-apitools<0.5.32,>=0.5.31->apache-beam[gcp]->-r feedstock/requirements.txt (line 5))
Using cached six-1.16.0-py2.py3-none-any.whl.metadata (1.8 kB)
Collecting pyasn1-modules>=0.2.1 (from google-auth>=1.2->gcsfs->-r feedstock/requirements.txt (line 4))
Using cached pyasn1_modules-0.4.0-py3-none-any.whl.metadata (3.4 kB)
Collecting rsa<5,>=3.1.4 (from google-auth>=1.2->gcsfs->-r feedstock/requirements.txt (line 4))
Using cached rsa-4.9-py3-none-any.whl.metadata (4.2 kB)
Collecting google-cloud-resource-manager<3.0.0dev,>=1.3.3 (from google-cloud-aiplatform<2.0,>=1.26.0->apache-beam[gcp]->-r feedstock/requirements.txt (line 5))
Using cached google_cloud_resource_manager-1.12.3-py2.py3-none-any.whl.metadata (5.3 kB)
Collecting shapely<3.0.0dev (from google-cloud-aiplatform<2.0,>=1.26.0->apache-beam[gcp]->-r feedstock/requirements.txt (line 5))
Using cached shapely-2.0.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (7.0 kB)
Collecting google-resumable-media<3.0dev,>=0.6.0 (from google-cloud-bigquery<4,>=2.0.0->apache-beam[gcp]->-r feedstock/requirements.txt (line 5))
Using cached google_resumable_media-2.7.0-py2.py3-none-any.whl.metadata (2.2 kB)
Collecting grpc-google-iam-v1<1.0.0dev,>=0.12.4 (from google-cloud-bigtable<3,>=2.19.0->apache-beam[gcp]->-r feedstock/requirements.txt (line 5))
Using cached grpc_google_iam_v1-0.13.0-py2.py3-none-any.whl.metadata (3.3 kB)
Collecting grpcio-status>=1.33.2 (from google-cloud-pubsub<3,>=2.1.0->apache-beam[gcp]->-r feedstock/requirements.txt (line 5))
Using cached grpcio_status-1.62.1-py3-none-any.whl.metadata (1.3 kB)
Collecting overrides<8.0.0,>=6.0.1 (from google-cloud-pubsublite<2,>=1.2.0->apache-beam[gcp]->-r feedstock/requirements.txt (line 5))
Using cached overrides-7.7.0-py3-none-any.whl.metadata (5.8 kB)
Collecting sqlparse>=0.4.4 (from google-cloud-spanner<4,>=3.0.0->apache-beam[gcp]->-r feedstock/requirements.txt (line 5))
Using cached sqlparse-0.4.4-py3-none-any.whl.metadata (4.0 kB)
Collecting deprecated>=1.2.14 (from google-cloud-spanner<4,>=3.0.0->apache-beam[gcp]->-r feedstock/requirements.txt (line 5))
Using cached Deprecated-1.2.14-py2.py3-none-any.whl.metadata (5.4 kB)
Collecting grpc-interceptor>=0.15.4 (from google-cloud-spanner<4,>=3.0.0->apache-beam[gcp]->-r feedstock/requirements.txt (line 5))
Using cached grpc_interceptor-0.15.4-py3-none-any.whl.metadata (8.4 kB)
Collecting google-crc32c<2.0dev,>=1.0 (from google-cloud-storage->gcsfs->-r feedstock/requirements.txt (line 4))
Using cached google_crc32c-1.5.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (2.3 kB)
Collecting docopt (from hdfs<3.0.0,>=2.1.0->apache-beam[gcp]->-r feedstock/requirements.txt (line 5))
Using cached docopt-0.6.2-py2.py3-none-any.whl
Collecting pyparsing!=3.0.0,!=3.0.1,!=3.0.2,!=3.0.3,<4,>=2.4.2 (from httplib2<0.23.0,>=0.8->apache-beam[gcp]->-r feedstock/requirements.txt (line 5))
Using cached pyparsing-3.1.2-py3-none-any.whl.metadata (5.1 kB)
Collecting tzlocal>=1.2 (from js2py<1,>=0.74->apache-beam[gcp]->-r feedstock/requirements.txt (line 5))
Using cached tzlocal-5.2-py3-none-any.whl.metadata (7.8 kB)
Collecting pyjsparser>=2.5.1 (from js2py<1,>=0.74->apache-beam[gcp]->-r feedstock/requirements.txt (line 5))
Using cached pyjsparser-2.7.1-py3-none-any.whl
Collecting jsonschema-specifications>=2023.03.6 (from jsonschema<5.0.0,>=4.0.0->apache-beam[gcp]->-r feedstock/requirements.txt (line 5))
Using cached jsonschema_specifications-2023.12.1-py3-none-any.whl.metadata (3.0 kB)
Collecting referencing>=0.28.4 (from jsonschema<5.0.0,>=4.0.0->apache-beam[gcp]->-r feedstock/requirements.txt (line 5))
Using cached referencing-0.34.0-py3-none-any.whl.metadata (2.8 kB)
Collecting rpds-py>=0.7.1 (from jsonschema<5.0.0,>=4.0.0->apache-beam[gcp]->-r feedstock/requirements.txt (line 5))
Using cached rpds_py-0.18.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.1 kB)
Collecting ujson (from kerchunk>=0.0.7->pangeo-forge-recipes==0.10.6->-r feedstock/requirements.txt (line 2))
Using cached ujson-5.9.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (8.7 kB)
Collecting dnspython<3.0.0,>=1.16.0 (from pymongo<5.0.0,>=3.8.0->apache-beam[gcp]->-r feedstock/requirements.txt (line 5))
Using cached dnspython-2.6.1-py3-none-any.whl.metadata (5.8 kB)
Collecting charset-normalizer<4,>=2 (from requests->pangeo-forge-esgf==0.1.1->-r feedstock/requirements.txt (line 1))
Using cached charset_normalizer-3.3.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (33 kB)
Collecting idna<4,>=2.5 (from requests->pangeo-forge-esgf==0.1.1->-r feedstock/requirements.txt (line 1))
Using cached idna-3.6-py3-none-any.whl.metadata (9.9 kB)
Collecting urllib3<3,>=1.21.1 (from requests->pangeo-forge-esgf==0.1.1->-r feedstock/requirements.txt (line 1))
Using cached urllib3-2.2.1-py3-none-any.whl.metadata (6.4 kB)
Collecting certifi>=2017.4.17 (from requests->pangeo-forge-esgf==0.1.1->-r feedstock/requirements.txt (line 1))
Using cached certifi-2024.2.2-py3-none-any.whl.metadata (2.2 kB)
Collecting pandas>=1.5 (from xarray>=0.18.0->pangeo-forge-recipes==0.10.6->-r feedstock/requirements.txt (line 2))
Using cached pandas-2.2.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (19 kB)
Collecting asciitree (from zarr>=2.12.0->pangeo-forge-recipes==0.10.6->-r feedstock/requirements.txt (line 2))
Using cached asciitree-0.3.3-py3-none-any.whl
Collecting cramjam>=2.3 (from fastparquet->pangeo-forge-recipes==0.10.6->-r feedstock/requirements.txt (line 2))
Using cached cramjam-2.8.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.2 kB)
Collecting requests-oauthlib>=0.7.0 (from google-auth-oauthlib->gcsfs->-r feedstock/requirements.txt (line 4))
Using cached requests_oauthlib-2.0.0-py2.py3-none-any.whl.metadata (11 kB)
Collecting wrapt<2,>=1.10 (from deprecated>=1.2.14->google-cloud-spanner<4,>=3.0.0->apache-beam[gcp]->-r feedstock/requirements.txt (line 5))
Using cached wrapt-1.16.0-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.6 kB)
Collecting zipp>=0.5 (from importlib-metadata>=4.13.0->dask>=2021.11.2->pangeo-forge-recipes==0.10.6->-r feedstock/requirements.txt (line 2))
Using cached zipp-3.18.1-py3-none-any.whl.metadata (3.5 kB)
Collecting pyasn1>=0.1.7 (from oauth2client>=1.4.12->google-apitools<0.5.32,>=0.5.31->apache-beam[gcp]->-r feedstock/requirements.txt (line 5))
Using cached pyasn1-0.6.0-py2.py3-none-any.whl.metadata (8.3 kB)
Collecting tzdata>=2022.7 (from pandas>=1.5->xarray>=0.18.0->pangeo-forge-recipes==0.10.6->-r feedstock/requirements.txt (line 2))
Using cached tzdata-2024.1-py2.py3-none-any.whl.metadata (1.4 kB)
Collecting locket (from partd>=1.2.0->dask>=2021.11.2->pangeo-forge-recipes==0.10.6->-r feedstock/requirements.txt (line 2))
Using cached locket-1.0.0-py2.py3-none-any.whl.metadata (2.8 kB)
Collecting oauthlib>=3.0.0 (from requests-oauthlib>=0.7.0->google-auth-oauthlib->gcsfs->-r feedstock/requirements.txt (line 4))
Using cached oauthlib-3.2.2-py3-none-any.whl.metadata (7.5 kB)
Using cached pangeo_forge_esgf-0.1.1-py3-none-any.whl (15 kB)
Using cached pangeo_forge_recipes-0.10.6-py3-none-any.whl (130 kB)
Using cached dynamic_chunks-0.0.2-py3-none-any.whl (12 kB)
Using cached gcsfs-2024.3.1-py2.py3-none-any.whl (34 kB)
Using cached fsspec-2024.3.1-py3-none-any.whl (171 kB)
Using cached aiohttp-3.9.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)
Using cached cachetools-5.3.3-py3-none-any.whl (9.3 kB)
Using cached cloudpickle-2.2.1-py3-none-any.whl (25 kB)
Using cached dask-2024.3.1-py3-none-any.whl (1.2 MB)
Using cached decorator-5.1.1-py3-none-any.whl (9.1 kB)
Using cached fastavro-1.9.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.3 MB)
Using cached fasteners-0.19-py3-none-any.whl (18 kB)
Using cached google_api_core-2.18.0-py3-none-any.whl (138 kB)
Using cached google_auth-2.29.0-py2.py3-none-any.whl (189 kB)
Using cached google_auth_httplib2-0.1.1-py2.py3-none-any.whl (9.3 kB)
Using cached google_cloud_aiplatform-1.44.0-py2.py3-none-any.whl (4.2 MB)
Using cached google_cloud_bigquery-3.19.0-py2.py3-none-any.whl (232 kB)
Using cached google_cloud_bigquery_storage-2.24.0-py2.py3-none-any.whl (190 kB)
Using cached google_cloud_bigtable-2.23.0-py2.py3-none-any.whl (357 kB)
Using cached google_cloud_core-2.4.1-py2.py3-none-any.whl (29 kB)
Using cached google_cloud_datastore-2.19.0-py2.py3-none-any.whl (176 kB)
Using cached google_cloud_dlp-3.16.0-py2.py3-none-any.whl (167 kB)
Using cached google_cloud_language-2.13.3-py2.py3-none-any.whl (143 kB)
Using cached google_cloud_pubsub-2.21.0-py2.py3-none-any.whl (273 kB)
Using cached google_cloud_pubsublite-1.9.0-py2.py3-none-any.whl (287 kB)
Using cached google_cloud_recommendations_ai-0.10.10-py2.py3-none-any.whl (180 kB)
Using cached google_cloud_spanner-3.44.0-py2.py3-none-any.whl (357 kB)
Using cached google_cloud_storage-2.16.0-py2.py3-none-any.whl (125 kB)
Using cached google_cloud_videointelligence-2.13.3-py2.py3-none-any.whl (240 kB)
Using cached google_cloud_vision-3.7.2-py2.py3-none-any.whl (459 kB)
Using cached grpcio-1.62.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.5 MB)
Using cached h5py-3.10.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.8 MB)
Using cached hdfs-2.7.0-py3-none-any.whl (34 kB)
Using cached httplib2-0.22.0-py3-none-any.whl (96 kB)
Using cached Js2Py-0.74-py3-none-any.whl (1.0 MB)
Using cached jsonpickle-3.0.3-py3-none-any.whl (40 kB)
Using cached jsonschema-4.21.1-py3-none-any.whl (85 kB)
Using cached kerchunk-0.2.4-py3-none-any.whl (3.3 MB)
Using cached numcodecs-0.12.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (7.9 MB)
Using cached numpy-1.26.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (18.3 MB)
Using cached objsize-0.7.0-py3-none-any.whl (11 kB)
Using cached orjson-3.9.15-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (138 kB)
Using cached packaging-24.0-py3-none-any.whl (53 kB)
Using cached proto_plus-1.23.0-py3-none-any.whl (48 kB)
Using cached protobuf-4.25.3-cp37-abi3-manylinux2014_x86_64.whl (294 kB)
Using cached pyarrow-14.0.2-cp311-cp311-manylinux_2_28_x86_64.whl (38.0 MB)
Using cached pyarrow_hotfix-0.6-py3-none-any.whl (7.9 kB)
Using cached pydot-1.4.2-py2.py3-none-any.whl (21 kB)
Using cached pymongo-4.6.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (680 kB)
Using cached python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB)
Using cached pytz-2024.1-py2.py3-none-any.whl (505 kB)
Using cached regex-2023.12.25-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (785 kB)
Using cached requests-2.31.0-py3-none-any.whl (62 kB)
Using cached typing_extensions-4.10.0-py3-none-any.whl (33 kB)
Using cached xarray-2024.2.0-py3-none-any.whl (1.1 MB)
Using cached zarr-2.17.1-py3-none-any.whl (207 kB)
Using cached zstandard-0.22.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.4 MB)
Using cached apache_beam-2.55.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (15.8 MB)
Using cached backoff-2.2.1-py3-none-any.whl (15 kB)
Using cached cftime-1.6.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.4 MB)
Using cached fastparquet-2024.2.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.8 MB)
Using cached google_auth_oauthlib-1.2.0-py2.py3-none-any.whl (24 kB)
Using cached h5netcdf-1.3.0-py3-none-any.whl (43 kB)
Using cached netCDF4-1.6.5-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.5 MB)
Using cached tqdm-4.66.2-py3-none-any.whl (78 kB)
Using cached aiosignal-1.3.1-py3-none-any.whl (7.6 kB)
Using cached attrs-23.2.0-py3-none-any.whl (60 kB)
Using cached certifi-2024.2.2-py3-none-any.whl (163 kB)
Using cached charset_normalizer-3.3.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (140 kB)
Using cached click-8.1.7-py3-none-any.whl (97 kB)
Using cached cramjam-2.8.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.0 MB)
Using cached Deprecated-1.2.14-py2.py3-none-any.whl (9.6 kB)
Using cached dnspython-2.6.1-py3-none-any.whl (307 kB)
Using cached frozenlist-1.4.1-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (272 kB)
Using cached google_cloud_resource_manager-1.12.3-py2.py3-none-any.whl (333 kB)
Using cached google_crc32c-1.5.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (32 kB)
Using cached google_resumable_media-2.7.0-py2.py3-none-any.whl (80 kB)
Using cached googleapis_common_protos-1.63.0-py2.py3-none-any.whl (229 kB)
Using cached grpc_google_iam_v1-0.13.0-py2.py3-none-any.whl (25 kB)
Using cached grpc_interceptor-0.15.4-py3-none-any.whl (20 kB)
Using cached grpcio_status-1.62.1-py3-none-any.whl (14 kB)
Using cached idna-3.6-py3-none-any.whl (61 kB)
Using cached importlib_metadata-7.1.0-py3-none-any.whl (24 kB)
Using cached jsonschema_specifications-2023.12.1-py3-none-any.whl (18 kB)
Using cached multidict-6.0.5-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (128 kB)
Using cached oauth2client-4.1.3-py2.py3-none-any.whl (98 kB)
Using cached overrides-7.7.0-py3-none-any.whl (17 kB)
Using cached pandas-2.2.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (13.0 MB)
Using cached partd-1.4.1-py3-none-any.whl (18 kB)
Using cached pyasn1_modules-0.4.0-py3-none-any.whl (181 kB)
Using cached pyparsing-3.1.2-py3-none-any.whl (103 kB)
Using cached PyYAML-6.0.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (757 kB)
Using cached referencing-0.34.0-py3-none-any.whl (26 kB)
Using cached requests_oauthlib-2.0.0-py2.py3-none-any.whl (24 kB)
Using cached rpds_py-0.18.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.1 MB)
Using cached rsa-4.9-py3-none-any.whl (34 kB)
Using cached shapely-2.0.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.5 MB)
Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Using cached sqlparse-0.4.4-py3-none-any.whl (41 kB)
Using cached toolz-0.12.1-py3-none-any.whl (56 kB)
Using cached tzlocal-5.2-py3-none-any.whl (17 kB)
Using cached urllib3-2.2.1-py3-none-any.whl (121 kB)
Using cached yarl-1.9.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (328 kB)
Using cached ujson-5.9.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (53 kB)
Using cached oauthlib-3.2.2-py3-none-any.whl (151 kB)
Using cached pyasn1-0.6.0-py2.py3-none-any.whl (85 kB)
Using cached tzdata-2024.1-py2.py3-none-any.whl (345 kB)
Using cached wrapt-1.16.0-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (80 kB)
Using cached zipp-3.18.1-py3-none-any.whl (8.2 kB)
Using cached locket-1.0.0-py2.py3-none-any.whl (4.4 kB)
Installing collected packages: pytz, pyjsparser, docopt, crcmod, asciitree, zstandard, zipp, wrapt, urllib3, ujson, tzlocal, tzdata, typing-extensions, tqdm, toolz, sqlparse, six, rpds-py, regex, pyyaml, pyparsing, pyasn1, pyarrow-hotfix, protobuf, packaging, overrides, orjson, objsize, oauthlib, numpy, multidict, locket, jsonpickle, idna, grpcio, google-crc32c, fsspec, frozenlist, fasteners, fastavro, dnspython, dill, decorator, cramjam, cloudpickle, click, charset-normalizer, certifi, cachetools, backoff, attrs, yarl, shapely, rsa, requests, referencing, python-dateutil, pymongo, pydot, pyasn1-modules, pyarrow, proto-plus, partd, numcodecs, js2py, importlib-metadata, httplib2, h5py, grpc-interceptor, googleapis-common-protos, google-resumable-media, deprecated, cftime, aiosignal, zarr, requests-oauthlib, pandas, oauth2client, netcdf4, jsonschema-specifications, hdfs, h5netcdf, grpcio-status, google-auth, dask, aiohttp, xarray, pangeo-forge-esgf, kerchunk, jsonschema, grpc-google-iam-v1, google-auth-oauthlib, google-auth-httplib2, google-apitools, google-api-core, fastparquet, google-cloud-core, dynamic-chunks, apache-beam, pangeo-forge-recipes, google-cloud-vision, google-cloud-videointelligence, google-cloud-storage, google-cloud-spanner, google-cloud-resource-manager, google-cloud-recommendations-ai, google-cloud-pubsub, google-cloud-language, google-cloud-dlp, google-cloud-datastore, google-cloud-bigtable, google-cloud-bigquery-storage, google-cloud-bigquery, google-cloud-pubsublite, google-cloud-aiplatform, gcsfs
Successfully installed aiohttp-3.9.3 aiosignal-1.3.1 apache-beam-2.55.0 asciitree-0.3.3 attrs-23.2.0 backoff-2.2.1 cachetools-5.3.3 certifi-2024.2.2 cftime-1.6.3 charset-normalizer-3.3.2 click-8.1.7 cloudpickle-2.2.1 cramjam-2.8.3 crcmod-1.7 dask-2024.3.1 decorator-5.1.1 deprecated-1.2.14 dill-0.3.1.1 dnspython-2.6.1 docopt-0.6.2 dynamic-chunks-0.0.2 fastavro-1.9.4 fasteners-0.19 fastparquet-2024.2.0 frozenlist-1.4.1 fsspec-2024.3.1 gcsfs-2024.3.1 google-api-core-2.18.0 google-apitools-0.5.31 google-auth-2.29.0 google-auth-httplib2-0.1.1 google-auth-oauthlib-1.2.0 google-cloud-aiplatform-1.44.0 google-cloud-bigquery-3.19.0 google-cloud-bigquery-storage-2.24.0 google-cloud-bigtable-2.23.0 google-cloud-core-2.4.1 google-cloud-datastore-2.19.0 google-cloud-dlp-3.16.0 google-cloud-language-2.13.3 google-cloud-pubsub-2.21.0 google-cloud-pubsublite-1.9.0 google-cloud-recommendations-ai-0.10.10 google-cloud-resource-manager-1.12.3 google-cloud-spanner-3.44.0 google-cloud-storage-2.16.0 google-cloud-videointelligence-2.13.3 google-cloud-vision-3.7.2 google-crc32c-1.5.0 google-resumable-media-2.7.0 googleapis-common-protos-1.63.0 grpc-google-iam-v1-0.13.0 grpc-interceptor-0.15.4 grpcio-1.62.1 grpcio-status-1.62.1 h5netcdf-1.3.0 h5py-3.10.0 hdfs-2.7.0 httplib2-0.22.0 idna-3.6 importlib-metadata-7.1.0 js2py-0.74 jsonpickle-3.0.3 jsonschema-4.21.1 jsonschema-specifications-2023.12.1 kerchunk-0.2.4 locket-1.0.0 multidict-6.0.5 netcdf4-1.6.5 numcodecs-0.12.1 numpy-1.26.4 oauth2client-4.1.3 oauthlib-3.2.2 objsize-0.7.0 orjson-3.9.15 overrides-7.7.0 packaging-24.0 pandas-2.2.1 pangeo-forge-esgf-0.1.1 pangeo-forge-recipes-0.10.6 partd-1.4.1 proto-plus-1.23.0 protobuf-4.25.3 pyarrow-14.0.2 pyarrow-hotfix-0.6 pyasn1-0.6.0 pyasn1-modules-0.4.0 pydot-1.4.2 pyjsparser-2.7.1 pymongo-4.6.2 pyparsing-3.1.2 python-dateutil-2.9.0.post0 pytz-2024.1 pyyaml-6.0.1 referencing-0.34.0 regex-2023.12.25 requests-2.31.0 requests-oauthlib-2.0.0 rpds-py-0.18.0 rsa-4.9 shapely-2.0.3 six-1.16.0 sqlparse-0.4.4 toolz-0.12.1 tqdm-4.66.2 typing-extensions-4.10.0 tzdata-2024.1 tzlocal-5.2 ujson-5.9.0 urllib3-2.2.1 wrapt-1.16.0 xarray-2024.2.0 yarl-1.9.4 zarr-2.17.1 zipp-3.18.1 zstandard-0.22.0
Using Any for unsupported type: typing.Sequence[~T]
Target Storage is FSSpecTarget(LocalFileSystem(, root_path="/home/jovyan/AREAS/PGF/hanging_bug/output/{job_name}/output")
Input Cache Storage is CacheFSSpecTarget(LocalFileSystem(, root_path="/home/jovyan/AREAS/PGF/hanging_bug/output/cache/") Parsing recipes... ==================== <function annotate_downstream_side_inputs at 0x7f418b674720> ==================== So this 'finishes' but the output looks wrong! 
There are 4! output chunks. 
Digging a bit further into the log output from above I see something that Nathan and I were wondering about earlier: If I read this correctly this means that this is not actually combining any of the (sub)fragments, since it only contains a single fragment to begin with. This should not happen in this case. I drew up a sketch of what I expecte to happen in this case I think that the grouping (of exactly two fragments, which are different in What seems to happen instead is that it writes the smallest unit (is this a fragment or subfragment - I am very confused by the terminology) as chunks to the target zarr store. This will results in a TON of writes of tiny chunks in certain scenarios which I guess could at least explain why jobs are slow and maybe how the risk of some random hang gets exponentially increased? phew this was a long one, but in summary I think that something is broken in the groupby/combine_fragments logic which does not actually produce the correct groups/lists/tuples of fragments that should be combined in an output chunk. |
|
nice write up and graphics @jbusecke 🥳 , well explained |
|
Nice digging Julius!
The logic these functions is unit-tested relatively thoroughly in test_rechunking.py. If you think there is a bug here, the next step would be to eliminate the complexity of the recipe execution and just isolate the logical error via a unit test. In general, we try to follow the beam-recommended practices for our PTtransforms
That means we shouldn't have to run beam code at all to debug and diagnose the sort of issue you're describing. You should be able to take your toy example and turn it into a simple test. Happy to help with this. |
Also, FWIW, the rechunking algorithm in Pangeo Forge is really not optimized for this extreme case. It means basically splitting up every pixel of a dataset into a individual Xarray Dataset. However, it should definitely work, even if the performance is very poor. |
This is the expected behavior for Have you tried explicitly specifying the time chunks, i.e. |
Oh yikes! Then I might have jumped down the wrong rabbithole😱. Ill double check this asap. In any case let me roll back some of the strong language above, since I think I might habe been overly immersed in my dynamic chunking function (CMIP) world where the behavior is to return len(dim) for any dimension that is not specified. Faster subway, I need to find out hahaha. |
|
Ohhh damn, @rabernat was right. I was making a wrong assumption here. Testing with Well that is very good (in the sense that we did not have a big bug in recipes!) but also sets us back in finding the reason for this bug. The only thing I think I can say at this point is that the issue seems to be somehow triggered by producing a lot of fragments or grouping them (since these jobs seems to generally get stuck on a read not a write I am less suspicious of the steps after generating / combining fragments). Welp, onwards...tomorrow. |
|
FWIW I think it would be helpful to keep working on a schematic like the one above and label the following terms in there:
I might have missed/double counted some terms, but having those enumerated might help with future debugging/refactoring. |
Point well taken. I think that none of my dynamic rechunkings will ever be this extreme, but a full 'pixel-level' rechunk should still work as you say, and stress testing this might be helpful here to reveal the (still mysterious) underlying reason for my stalls. |
|
I think the ultimate point of confusion is the logger entry coming from this line. Am I wrong to assume that this should receive all the fragments that should 'land' in a given target chunks, and thus in our case here It should display more than one fragment to actually combine? There is basically a special case where all the rechunking does is 'splitting' into smaller fragment and then each one is written into a target chunk, but that should not be the case here (anymore 😆). For my little toy example (updated version with explicit chunks) I get: Which I interpret as only a single fragment? This would be helpful to get some clarification on before I continue on this. |
That is the correct interpretation. There is a list of fragments of length 1. Is that from the previous example ( |
|
It is from the updated version with explicit chunks for all existing dimensions! We saw similar one fragment lists in the more complex recipes (even though I still need to double check there to exclude my error from above). Seeing only a single fragment in there is what led me down the path of making the toy example above in the first place. Ill look further into the keys and grouping when I have the time. |
But then how is the dataset being written correctly? Could you show the full logging output? I'd still encourage you to remove the PTransforms and Pipelines from the mix to focus on the core rechunking logic, which is quite simple: pangeo-forge-recipes/pangeo_forge_recipes/transforms.py Lines 395 to 402 in 55fff0b |
|
@rabernat can you confirm that the basic assumption here is correct: In the case sketched out above + target_chunks={'x':1, 'time':4} we would expect a list of 2 fragments to be combined to be shown in that log statement? |
Not without a simpler reproducer that just involves the functions |
|
I am sorry there must be something clearly wrong with the way I am testing this locally (on the LEAP hub). I just reran the example and we get the expected (2 fragement) logging call:
I am very sorry for all the noise and loss of focus. I think I have convinced myself that the rechunking logic works as intended here and that the issue lies elsewhere. |
|
I just wanted to reset and confirm that my (still complex) reproducer recipe still hangs! The pertaining log output is here Which I interpret as being stuck on reading something...would love to hear how others think about this. Does this eagerly load the data to be combined, or just read the metadata at this point? Is the issue that we are trying to read the same metadata from 'too many' processes? Or do we simply need to allow fsspec to fail and retry? Disable caching? One more possibly important piece of information here is that this does not seem to be just a single read being hung up. At least that is how I interpret all of these log messages that have a similar content as above but different 'bundles': The frustrating thing about being back at this end of the equation is that this example only fails on dataflow (@ranchodeluxe was able to run it on flink, and I ran it locally) making the debugging very very hard. I would love nothing more than to really dig deeper into this today but have too much other things on my plate unfortunately. Maybe a little breather is also appropriate, this issue is a frustrating one. Very curious if others have ideas how to proceed. |
|
Just a quick update here:
|
|
Ok so some preliminary conclusions: At least for my case, setting I was able to get around the issues with the local disk by specifying a larger attached disk size (see runner PR), but now instead of errors I am seeing all the usual stalling behavior again:
I am not sure if this is the same issue of deadlocking in fsspec? But unfortunately for now it seems like EDIT: As per convo with @rabernat over in #722 i tried setting both This one sounds curious: Def out of my wheelhouse, but maybe @ranchodeluxe or @moradology have an idea? |
|
Yeah, this is one of the things that can make it difficult in the extreme to debug distributed systems. The seemingly meaningless trace here is a symptom of the generality and the abstraction that is at the heart of their execution model. Your functions and data have to get passed around inside something capable of executing them in various contexts. A rough translation of what this looks like to me is: 'status was set to cancelled while in the process of iterating through the data we need to operate over'. Could well be there's an underlying trace further up or swallowed but I wouldn't be shocked to see this as output from killing the master process |
As discussed in the last community meeting and on the call with @moradology and @ranchodeluxe today, I will try to boil down one of the CMIP6 feedstock examples that I observed to 'hang' (not completing within several hours, even though the caching was already complete!).
The dataset unique identifier is
'CMIP6.CMIP.CMCC.CMCC-ESM2.historical.r1i1p1f1.3hr.pr.gn.v20210114'and the file urls are:A stripped down recipe to reproduce this could look like this:
(to be revised, just leaving this here as a starting point)
@ranchodeluxe do you mind sending me the steps you use to run this on a jupyterhub?
Will fill this in as I go along.
The text was updated successfully, but these errors were encountered: