Configure the pyMBE virtual environment #12
Conversation
| ${ESPResSo_build_path}/pypresso your_simulation_script.py | ||
| source pymbe/bin/activate | ||
| python3 -m pip install ipykernel "jupyterlab>=4.0.8" "PyOpenGL>=3.1.5" | ||
| python3 -m ipykernel install --user --name=pyMBE |
There was a problem hiding this comment.
Note: I tried to encapsulate the pyMBE kernel into the virtual environment itself, but Jupyter only seems to detect the ones installed in the user's $HOME folder:
(pymbe) $ python3 -m ipykernel install --sys-prefix --name=pyMBE
Installed kernelspec pyMBE in /work/jgrad/pyMBE/pymbe/share/jupyter/kernels/pymbe
(pymbe) $ which jupyter
/work/jgrad/pyMBE/pymbe/bin/jupyter
(pymbe) $ jupyter kernelspec list
Available kernels:
pymbe /home/jgrad/.local/share/jupyter/kernels/pymbe
python3 /home/jgrad/.local/share/jupyter/kernels/python3
I couldn't find a way to detect the local folders. I tried to generate a default Jupyter config file in the virtual environment, thinking I could adapt something a path in there, but Jupyter ended up writing the default config file in my $HOME folder directly... So be careful if you want to tinker with Jupyter settings in the virtual environment.
There was a problem hiding this comment.
@jngrad I followed the instructions and it was pretty clear to me how to set up and use the new virtual environment of pyMBE. I fixed all paths in the scripts and removed the previous logic to import pyMBE.
I have to say that this new pyMBE environment is very cool and I think that it could be very helpful to make our lives easier to use both espresso and pyMBE. It is, however, not clear to me how to setup pyMBE when using it in a cluster when you do not now the path in which espresso was built. I tried to import the espresso module built-in in the cluster we use at NTNU and then following the instructions but I failed:
[pablb@idun-login1 pyMBE_jn]$ module load ESPResSo/4.2.1-foss-2022a
[pablb@idun-login1 pyMBE_jn]$ python3 -m venv pymbe
[pablb@idun-login1 pyMBE_jn]$ source pymbe/bin/activate
(pymbe) [pablb@idun-login1 pyMBE_jn]$ python3 -m pip install -r requirements.txt
Collecting numpy>=1.23
Downloading numpy-1.26.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (18.2 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 18.2/18.2 MB 82.3 MB/s eta 0:00:00
Collecting pandas>=1.5.3
Downloading pandas-2.2.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (13.0 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13.0/13.0 MB 65.9 MB/s eta 0:00:00
Requirement already satisfied: pint>=0.20.01 in /cluster/apps/eb/software/Pint/0.22-GCCcore-11.3.0/lib/python3.10/site-packages (from -r requirements.txt (line 3)) (0.22)
Collecting pint-pandas==0.5
Downloading Pint_Pandas-0.5-py3-none-any.whl (25 kB)
Collecting biopandas==0.5.1.dev0
Downloading biopandas-0.5.1.dev0-py3-none-any.whl (1.2 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 11.7 MB/s eta 0:00:00
Requirement already satisfied: scipy in /cluster/apps/eb/software/SciPy-bundle/2022.05-foss-2022a/lib/python3.10/site-packages (from -r requirements.txt (line 6)) (1.8.1)
Collecting matplotlib
Downloading matplotlib-3.8.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.6 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.6/11.6 MB 80.5 MB/s eta 0:00:00
Collecting tqdm
Downloading tqdm-4.66.2-py3-none-any.whl (78 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 78.3/78.3 KB 9.4 MB/s eta 0:00:00
Requirement already satisfied: setuptools in ./pymbe/lib/python3.10/site-packages (from biopandas==0.5.1.dev0->-r requirements.txt (line 5)) (58.1.0)
Collecting looseversion==1.1.2
Downloading looseversion-1.1.2-py3-none-any.whl (8.1 kB)
Collecting mmtf-python==1.1.3
Downloading mmtf_python-1.1.3-py2.py3-none-any.whl (25 kB)
Collecting msgpack>=1.0.0
Downloading msgpack-1.0.8-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (385 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 385.1/385.1 KB 39.1 MB/s eta 0:00:00
Collecting python-dateutil>=2.8.2
Downloading python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 229.9/229.9 KB 25.5 MB/s eta 0:00:00
Collecting pytz>=2020.1
Downloading pytz-2024.1-py2.py3-none-any.whl (505 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 505.5/505.5 KB 35.1 MB/s eta 0:00:00
Collecting tzdata>=2022.7
Downloading tzdata-2024.1-py2.py3-none-any.whl (345 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 345.4/345.4 KB 25.2 MB/s eta 0:00:00
Collecting typing-extensions
Downloading typing_extensions-4.10.0-py3-none-any.whl (33 kB)
Collecting numpy>=1.23
Downloading numpy-1.24.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.3 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17.3/17.3 MB 90.7 MB/s eta 0:00:00
Collecting pyparsing>=2.3.1
Downloading pyparsing-3.1.1-py3-none-any.whl (103 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 103.1/103.1 KB 12.1 MB/s eta 0:00:00
Collecting kiwisolver>=1.3.1
Downloading kiwisolver-1.4.5-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.6 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.6/1.6 MB 81.6 MB/s eta 0:00:00
Collecting packaging>=20.0
Downloading packaging-23.2-py3-none-any.whl (53 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 53.0/53.0 KB 5.7 MB/s eta 0:00:00
Collecting fonttools>=4.22.0
Downloading fonttools-4.49.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.6 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.6/4.6 MB 95.1 MB/s eta 0:00:00
Collecting pillow>=8
Downloading pillow-10.2.0-cp310-cp310-manylinux_2_28_x86_64.whl (4.5 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.5/4.5 MB 76.1 MB/s eta 0:00:00
Collecting cycler>=0.10
Downloading cycler-0.12.1-py3-none-any.whl (8.3 kB)
Collecting contourpy>=1.0.1
Downloading contourpy-1.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (310 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 310.7/310.7 KB 32.2 MB/s eta 0:00:00
Collecting six>=1.5
Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Installing collected packages: pytz, tzdata, typing-extensions, tqdm, six, pyparsing, pillow, packaging, numpy, msgpack, looseversion, kiwisolver, fonttools, cycler, python-dateutil, mmtf-python, contourpy, pandas, matplotlib, pint-pandas, biopandas
Attempting uninstall: numpy
Found existing installation: numpy 1.22.3
Not uninstalling numpy at /cluster/apps/eb/software/SciPy-bundle/2022.05-foss-2022a/lib/python3.10/site-packages, outside environment /cluster/home/pablb/pymBE_jn/pymbe
Can't uninstall 'numpy'. No files were found to uninstall.
Attempting uninstall: pandas
Found existing installation: pandas 1.4.2
Not uninstalling pandas at /cluster/apps/eb/software/SciPy-bundle/2022.05-foss-2022a/lib/python3.10/site-packages, outside environment /cluster/home/pablb/pymBE_jn/pymbe
Can't uninstall 'pandas'. No files were found to uninstall.
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
nbconvert 6.5.3 requires jinja2>=3.0, which is not installed.
nbconvert 6.5.3 requires MarkupSafe>=2.0, which is not installed.
nbconvert 6.5.3 requires pygments>=2.4.1, which is not installed.
jupyter-packaging 0.12.0 requires tomlkit, which is not installed.
jupyter-packaging 0.12.0 requires wheel, which is not installed.
ipykernel 6.13.0 requires psutil, which is not installed.
bleach 5.0.1 requires webencodings, which is not installed.
argon2-cffi 20.1.0 requires cffi>=1.0.0, which is not installed.
jupyter-packaging 0.12.0 requires setuptools>=60.2.0, but you have setuptools 58.1.0 which is incompatible.
Successfully installed biopandas-0.5.1.dev0 contourpy-1.2.0 cycler-0.12.1 fonttools-4.49.0 kiwisolver-1.4.5 looseversion-1.1.2 matplotlib-3.8.3 mmtf-python-1.1.3 msgpack-1.0.8 numpy-1.24.4 packaging-23.2 pandas-2.2.1 pillow-10.2.0 pint-pandas-0.5 pyparsing-3.1.1 python-dateutil-2.9.0.post0 pytz-2024.1 six-1.16.0 tqdm-4.66.2 typing-extensions-4.10.0 tzdata-2024.1
WARNING: You are using pip version 22.0.4; however, version 24.0 is available.
You should consider upgrading via the '/cluster/home/pablb/pymBE_jn/pymbe/bin/python3 -m pip install --upgrade pip' command.
(pymbe) [pablb@idun-login1 pyMBE_jn]$ pypresso testsuite/LYS_ASP_peptide.py
Traceback (most recent call last):
File "/cluster/home/pablb/pyMBE_jn/testsuite/LYS_ASP_peptide.py", line 7, in <module>
from tqdm import tqdm
ModuleNotFoundError: No module named 'tqdm'
There was a problem hiding this comment.
I am too used to building ESPResSo myself on clusters 😀
The configure_venv.py now detects if ESPResSo is already available as a module, so that cluster users can module load ESPResSo. I had to adapt the argparse syntax, because positional arguments cannot be made optional. The configure_venv.py script is now tested in CI as well.
There is also a venv option to only install the subset of Python packages that are not already loaded with ESPResSo; this should avoid the broken Python dependency tree you observed on the cluster. This is now documented in the readme file too.
There was a problem hiding this comment.
@pm-blanco @jngrad, I was following the new instructions and I got this error while running
python3 maintainer/configure_venv.py --espresso_path=~/espresso/build " ...
(pymbe) student@student-Swift3:~/pyMBE$ python3 maintainer/configure_venv.py --espresso_path=/home/student/espresso-4.2.1/espresso/build
Traceback (most recent call last):
File "maintainer/configure_venv.py", line 29, in <module>
make_pth("pymbe", os.path.dirname(os.path.dirname(__file__)))
File "maintainer/configure_venv.py", line 13, in make_pth
raise ValueError(f"Folder '{path}' doesn't exist")
ValueError: Folder '' doesn't existIn order to check that the provided path was right, I did 'ls' to the folder and it normally showed the content.
There was a problem hiding this comment.
Could you please check if the issue persists after replacing __file__ by os.path.abspath(__file__) at line 29?
There was a problem hiding this comment.
I did it and it worked. Now, It complained when installing the requirements because it could not find a version that satisfies the requirement pint-pandas==0.5. While looking for an explanation, we realized that my python's version is 3.8 while Pablo's version is 3.10 (it runs without problems). Could it be that the previous error is related to this fact as well? If so then should be worthy to specify which is the minimum required version of python.
Requirement already satisfied: numpy>=1.23 in ./pymbe/lib/python3.8/site-packages (from -r requirements.txt (line 1)) (1.24.4)
Requirement already satisfied: pandas>=1.5.3 in ./pymbe/lib/python3.8/site-packages (from -r requirements.txt (line 2)) (2.0.3)
Requirement already satisfied: pint>=0.20.01 in ./pymbe/lib/python3.8/site-packages (from -r requirements.txt (line 3)) (0.21.1)
ERROR: Could not find a version that satisfies the requirement pint-pandas==0.5 (from -r requirements.txt (line 4)) (from versions: 0.1, 0.2, 0.3)
ERROR: No matching distribution found for pint-pandas==0.5 (from -r requirements.txt (line 4))There was a problem hiding this comment.
pint-pandas is not packaged for Python 3.8. It is probably related to NEP 29 - Recommend Python and Numpy version support as a community policy standard. Major Python packages like matplotlib, jupyter and scipy had already adopted NEP 29 in 2020 (around the time when pint-pandas got its first release). ESPResSo eventually adopted NEP 29 in 2022 because maintaining support for older Python versions was not worth the trouble. According to the NEP 29 drop schedule, Python 3.8 is no longer officially supprted by the NumPy ecosystem since early 2023, and they'll drop support for Python 3.9 next month.
|
I tested the virtual environment and I had two minor observations:
Other than that, the rest of the tutorial on how to set up the environment was very clear to me. |
…ften dependency on pint-pandas to 0.3
Fixes #10
Provide instructions to set up a cross-platform virtual environment where pyMBE and ESPResSo can be imported directly in the Python interpreter. Provide instructions to set up a Jupyter kernel that uses that environment.
The Makefile no longer hardcodes the path to the ESPResSo build folder. The sample scripts and notebooks no longer need to detect where the pyMBE library is located.