Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add apache-beam to pangeo-notebook #331

Merged
merged 2 commits into from
May 26, 2022
Merged

add apache-beam to pangeo-notebook #331

merged 2 commits into from
May 26, 2022

Conversation

jhamman
Copy link
Member

@jhamman jhamman commented May 24, 2022

This PR adds apache-beam to the Pangeo-Notebook.

xref: pangeo-forge/pangeo-forge-recipes#256, #328

@github-actions
Copy link
Contributor

Binder 👈 Try on Mybinder.org!
Binder 👈 Try on Pangeo GCP Binder!
Binder 👈 Try on Pangeo AWS Binder!

@pangeo-bot
Copy link
Collaborator

/condalock
Automatically locking new conda environment, building, and testing images...

@jhamman
Copy link
Member Author

jhamman commented May 24, 2022

/condalock

@jhamman
Copy link
Member Author

jhamman commented May 24, 2022

Not sure what is going on here. The last lock failed with this error:

Encountered problems while solving:
  - nothing provides openssl >=1.1.1,<1.1.2.0a0 needed by python-2.7.15-h9bab390_2

Earlier I got this error:

Encountered problems while solving:
  - package xcape-0.1-py36h7a746c1_0 requires python_abi 3.6.* *_cp36m, but none of the providers can be installed

@scottyhq
Copy link
Member

If there is an incompatible set of packages, the conda/mamba messages can be really confusing trying all sorts of older versions and apparently different python versions even if it's pinned to 3.9*.... A strategy for these situations it to explicitly pin the package you're trying to add to its most recent version (- apache-beam=2.38.0=py39hb9d737c_1), then lock.

  1. We then find package intake-stac-0.2.0-py_0 requires sat-stac, but none of the providers can be installed...

  2. There is often an incompatible pin in the required package (sat-stac) and the one you're installing (apache-beam): mamba create -n tmp apache-beam=2.38.0=py39hb9d737c_1 sat-stac3.

Encountered problems while solving:
  - package sat-stac-0.1.3-py_0 requires python-dateutil >=2.7.5,<2.8, but none of the providers can be installed
  1. Look at the dependencies for apache-beam conda search --info apache-beam=2.38.0=py39hb9d737c_1
    - python-dateutil >=2.8.0,<3

@scottyhq
Copy link
Member

So oddly enough... @jhamman we need to update the intake-stac conda-forge feedstock which is out of sync with requirements.txt

@jhamman
Copy link
Member Author

jhamman commented May 26, 2022

/condalock

@jhamman jhamman merged commit 39778b5 into pangeo-data:master May 26, 2022
@jhamman jhamman deleted the feature/beam branch May 26, 2022 13:21
Comment on lines -369 to +361
https://conda.anaconda.org/conda-forge/linux-64/arrow-cpp-8.0.0-py39heccc63a_1_cpu.tar.bz2#300ebd380aae1fe0bbf04ff1424b042e
https://conda.anaconda.org/conda-forge/linux-64/arrow-cpp-6.0.1-py39h15c4769_19_cpu.tar.bz2#049c51026ed6a8a9a02a3ae27e07172e
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just noting seems like a big drop?

Copy link
Member Author

@jhamman jhamman May 26, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment on lines -495 to +486
https://conda.anaconda.org/conda-forge/linux-64/pyarrow-8.0.0-py39h42d110c_1_cpu.tar.bz2#9a6fd682c08e88c5b0e18ba84916b2b8
https://conda.anaconda.org/conda-forge/linux-64/pyqt-5.12.3-py39hf3d152e_8.tar.bz2#466425e3ee3b190e06b8a5a7098421aa
https://conda.anaconda.org/conda-forge/linux-64/pyarrow-6.0.1-py39h42d110c_19_cpu.tar.bz2#ce21bb03b644c34f5626a872fa393e2d
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note pyarrow drop from 8.0 -> 6.0, also pyqt is finally gone b/c of new parcels package!

@scottyhq
Copy link
Member

despite adding apache-beam here the image size actually dropped by 600MB b/c of removing pyqt in parcels 🎉

REPOSITORY               TAG         IMAGE ID       CREATED                  SIZE
pangeo/pangeo-notebook   PR          0cd447b9cf3c   Less than a second ago   3.33GB
pangeo/pangeo-notebook   PR          dcae122013df   Less than a second ago   3.94GB

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants