Fix #261 - pkg_resources migration #262

dnil · 2024-11-29T14:24:34Z

Description

Fixed

Refactored (mostly ChatGPT - thank you!) to use importlib
Refactored to use pathlib instead of os.path and path.py

I have aimed to have it safe for python >=3.9. As the PR is now, it might theoretically work back to 3.5/3.7 something, but also definitely with 3.13. More input on what we should support welcome. If we leave 3.7, we must update the Dockerfile as well, which currently runs an old 3.7 alpine. 🙄

How to prepare for test

Ssh to relevant server (depending on type of change)
Use stage: us
Paxa the environment: paxa

Install on stage (example for Hasta):

bash /home/proj/production/servers/resources/hasta.scilifelab.se/update-tool-stage.sh -e S_[TOOL]-t [TOOL] -b [THIS-BRANCH-NAME] -a

How to test

Do ...

Expected test outcome

Check that ...
Take a screenshot and attach or copy/paste the output.

Review

Tests executed by
"Merge and deploy" approved by
Thanks for filling in who performed the code review and the test!

This version is a

MAJOR - when you make incompatible API changes
MINOR - when you add functionality in a backwards compatible manner
PATCH - when you make backwards compatible bug fixes or documentation/instructions

Implementation Plan

Document in ...
Deploy this branch on ...
Inform to ...

dnil · 2024-12-04T12:32:41Z

Ok, tests pass at least, so time for some feedback! 😁

northwestwitch · 2024-12-04T12:48:02Z

If we leave 3.7, we must update the Dockerfile as well, which currently runs an old 3.7 alpine. 🙄

OMG!

northwestwitch · 2024-12-04T12:54:27Z

I think that since we are touching the code we should aim to support python >=3.9 only. Doesn't have to be in this PR, but we should update python both in the Dockerfile and in the actions. Is it a lot of refactoring in the code that you are introducing in this PR when we update python?

northwestwitch · 2024-12-04T12:59:36Z

We do have an image ready with python 3.11 and miniconda. I could work on one with python 3.12 and miniconda?

dnil · 2024-12-04T13:07:29Z

Sounds great! I can switch to the 3.11+miniconda here directly. It would ofcourse be possible to build (or brew) outside of conda, but that would be a drop in replacement!

northwestwitch

It looks good to me. Great refactoring! 💯

chanjo/__init__.py

northwestwitch · 2024-12-04T13:04:59Z

chanjo/cli/base.py

-import pkg_resources
+try:
+    from importlib.metadata import entry_points
+except ImportError:  # Backport support for importlib metadata on Python 3.7


Now I understand, it's for the Dockerfile!

It doesn't hurt much, other than perhaps that we install the compat libs without using them, so I guess we just keep it.

northwestwitch · 2024-12-04T13:07:28Z

chanjo/cli/base.py

@@ -16,14 +19,20 @@

 LOG = logging.getLogger(__name__)

+COMMAND_GROUP_KEY = 'chanjo.subcommands.4'


What is this subcommands.4 group? I'm not familiar with this syntax

As far as I understood it's just a label (

chanjo/setup.py

Line 96 in 1b2d859

'chanjo.subcommands.4': [

). I just kept it.

chanjo/cli/init.py

chanjo/init/demo.py

chanjo/testutils.py

tests/conftest.py

Co-authored-by: Chiara Rasi <rasi.chiara@gmail.com>

dnil · 2024-12-04T16:10:07Z

Deployed to stage:

[daniel.nilsson@hasta:~] [S_chanjo] $ bash /home/proj/production/servers/resources/hasta.scilifelab.se/update-tool-stage.sh -e S_chanjo -t chanjo -b migrate_pkg_resources
Collecting git+https://github.com/Clinical-Genomics/chanjo@migrate_pkg_resources
  Cloning https://github.com/Clinical-Genomics/chanjo (to revision migrate_pkg_resources) to ./tmp/pip-req-build-6e0m3uwv
  Running command git clone -q https://github.com/Clinical-Genomics/chanjo /home/daniel.nilsson/tmp/pip-req-build-6e0m3uwv
  Running command git checkout -b migrate_pkg_resources --track origin/migrate_pkg_resources
  Switched to a new branch 'migrate_pkg_resources'
  Branch migrate_pkg_resources set up to track remote branch migrate_pkg_resources from origin.
  Resolved https://github.com/Clinical-Genomics/chanjo to commit 9f5b2da3772e3a7d32d9a6da159330d6c9fe5f92
  Preparing metadata (setup.py) ... done
Building wheels for collected packages: chanjo
  Building wheel for chanjo (setup.py) ... done
  Created wheel for chanjo: filename=chanjo-4.7.1-py2.py3-none-any.whl size=45449 sha256=a6b202c5d89b35a1707c587ebec111db110acb408248b2d1a311fcf72bff89d7
  Stored in directory: /home/daniel.nilsson/tmp/pip-ephem-wheel-cache-v2582w4z/wheels/78/c4/b6/8b300039f6ac80d9363e289a87c79746b679491f9c46c8fa17
Successfully built chanjo
Installing collected packages: chanjo
  Attempting uninstall: chanjo
    Found existing installation: chanjo 4.7.1
    Uninstalling chanjo-4.7.1:
      Successfully uninstalled chanjo-4.7.1
Successfully installed chanjo-4.7.1
Collecting git+https://github.com/Clinical-Genomics/chanjo@migrate_pkg_resources
  Cloning https://github.com/Clinical-Genomics/chanjo (to revision migrate_pkg_resources) to ./tmp/pip-req-build-zbx6pqy8
  Running command git clone -q https://github.com/Clinical-Genomics/chanjo /home/daniel.nilsson/tmp/pip-req-build-zbx6pqy8
  Running command git checkout -b migrate_pkg_resources --track origin/migrate_pkg_resources
  Switched to a new branch 'migrate_pkg_resources'
  Branch migrate_pkg_resources set up to track remote branch migrate_pkg_resources from origin.
  Resolved https://github.com/Clinical-Genomics/chanjo to commit 9f5b2da3772e3a7d32d9a6da159330d6c9fe5f92
  Preparing metadata (setup.py) ... done
Requirement already satisfied: coloredlogs in /home/proj/stage/bin/miniconda3/envs/S_chanjo/lib/python3.11/site-packages (from chanjo==4.7.1) (15.0.1)
Requirement already satisfied: click in /home/proj/stage/bin/miniconda3/envs/S_chanjo/lib/python3.11/site-packages (from chanjo==4.7.1) (8.1.7)
Requirement already satisfied: cryptography in /home/proj/stage/bin/miniconda3/envs/S_chanjo/lib/python3.11/site-packages (from chanjo==4.7.1) (42.0.8)
Requirement already satisfied: toolz in /home/proj/stage/bin/miniconda3/envs/S_chanjo/lib/python3.11/site-packages (from chanjo==4.7.1) (0.12.1)
Requirement already satisfied: pyyaml in /home/proj/stage/bin/miniconda3/envs/S_chanjo/lib/python3.11/site-packages (from chanjo==4.7.1) (6.0.1)
Requirement already satisfied: importlib_metadata in /home/proj/stage/bin/miniconda3/envs/S_chanjo/lib/python3.11/site-packages (from chanjo==4.7.1) (7.1.0)
Requirement already satisfied: importlib_resources in /home/proj/stage/bin/miniconda3/envs/S_chanjo/lib/python3.11/site-packages (from chanjo==4.7.1) (6.4.5)
Requirement already satisfied: pymysql in /home/proj/stage/bin/miniconda3/envs/S_chanjo/lib/python3.11/site-packages (from chanjo==4.7.1) (1.1.1)
Requirement already satisfied: sqlalchemy>=2.0 in /home/proj/stage/bin/miniconda3/envs/S_chanjo/lib/python3.11/site-packages (from chanjo==4.7.1) (2.0.30)
Requirement already satisfied: sqlservice>=3.0 in /home/proj/stage/bin/miniconda3/envs/S_chanjo/lib/python3.11/site-packages (from chanjo==4.7.1) (3.0.0)
Requirement already satisfied: typing-extensions>=4.6.0 in /home/proj/stage/bin/miniconda3/envs/S_chanjo/lib/python3.11/site-packages (from sqlalchemy>=2.0->chanjo==4.7.1) (4.11.0)
Requirement already satisfied: greenlet!=0.4.17 in /home/proj/stage/bin/miniconda3/envs/S_chanjo/lib/python3.11/site-packages (from sqlalchemy>=2.0->chanjo==4.7.1) (3.0.3)
Requirement already satisfied: humanfriendly>=9.1 in /home/proj/stage/bin/miniconda3/envs/S_chanjo/lib/python3.11/site-packages (from coloredlogs->chanjo==4.7.1) (10.0)
Requirement already satisfied: cffi>=1.12 in /home/proj/stage/bin/miniconda3/envs/S_chanjo/lib/python3.11/site-packages (from cryptography->chanjo==4.7.1) (1.16.0)
Requirement already satisfied: zipp>=0.5 in /home/proj/stage/bin/miniconda3/envs/S_chanjo/lib/python3.11/site-packages (from importlib_metadata->chanjo==4.7.1) (3.18.2)
Requirement already satisfied: pycparser in /home/proj/stage/bin/miniconda3/envs/S_chanjo/lib/python3.11/site-packages (from cffi>=1.12->cryptography->chanjo==4.7.1) (2.22)
fatal: Not a git repository (or any parent up to mount point /home)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
repository is clean
Logging deploy ...
Getting deployer... done.
Getting last commit message and SHA... done.
Getting version of deploy scripts... /home/daniel.nilsson
done.
Log deploy... done.
chanjo, version 4.7.1

Testing a case:

[daniel.nilsson@hasta:~] [S_base] $ cg upload coverage magicaldog
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
----------------- UPLOAD -----------------
----------------- COVERAGE --------------------
Running command /home/proj/stage/bin/miniconda3/envs/S_chanjo/bin/chanjo --config /home/proj/stage/servers/config/hasta.scilifelab.se/chanjo-stage.yaml db samples -s ACC8033A7
Sample already loaded, deleting previous entry and re-uploading: ACC8033A7
Running command /home/proj/stage/bin/miniconda3/envs/S_chanjo/bin/chanjo --config /home/proj/stage/servers/config/hasta.scilifelab.se/chanjo-stage.yaml db remove ACC8033A7
Running command /home/proj/stage/bin/miniconda3/envs/S_chanjo/bin/chanjo --config /home/proj/stage/servers/config/hasta.scilifelab.se/chanjo-stage.yaml load --sample ACC8033A7 --name 21166-I-1A --group magicaldog --group-name 21166-trio --threshold 10 /home/proj/stage/housekeeper-bundles/magicaldog/2024-02-02/ACC8033A7_lanes_1234_sorted_md_coverage.bed
Running command /home/proj/stage/bin/miniconda3/envs/S_chanjo/bin/chanjo --config /home/proj/stage/servers/config/hasta.scilifelab.se/chanjo-stage.yaml db samples -s ACC13777A2
Sample already loaded, deleting previous entry and re-uploading: ACC13777A2
Running command /home/proj/stage/bin/miniconda3/envs/S_chanjo/bin/chanjo --config /home/proj/stage/servers/config/hasta.scilifelab.se/chanjo-stage.yaml db remove ACC13777A2
Running command /home/proj/stage/bin/miniconda3/envs/S_chanjo/bin/chanjo --config /home/proj/stage/servers/config/hasta.scilifelab.se/chanjo-stage.yaml load --sample ACC13777A2 --name 21166-II-1U --group magicaldog --group-name 21166-trio --threshold 10 /home/proj/stage/housekeeper-bundles/magicaldog/2024-02-02/ACC13777A2_lanes_12345678_sorted_md_coverage.bed
Running command /home/proj/stage/bin/miniconda3/envs/S_chanjo/bin/chanjo --config /home/proj/stage/servers/config/hasta.scilifelab.se/chanjo-stage.yaml db samples -s ACC13777A3
Sample already loaded, deleting previous entry and re-uploading: ACC13777A3
Running command /home/proj/stage/bin/miniconda3/envs/S_chanjo/bin/chanjo --config /home/proj/stage/servers/config/hasta.scilifelab.se/chanjo-stage.yaml db remove ACC13777A3
Running command /home/proj/stage/bin/miniconda3/envs/S_chanjo/bin/chanjo --config /home/proj/stage/servers/config/hasta.scilifelab.se/chanjo-stage.yaml load --sample ACC13777A3 --name 21166-II-2U --group magicaldog --group-name 21166-trio --threshold 10 /home/proj/stage/housekeeper-bundles/magicaldog/2024-02-02/ACC13777A3_lanes_12345678_sorted_md_coverage.bed

dnil added 14 commits November 29, 2024 15:17

Fix #261 - pkg_resources migration

514162d

first try the backport versions

f4ab437

make that a pre 3.10 fallback

2bbd6cf

workaround for pathlib instead of os.path

68a9b7f

one more abspath

434bbd7

mkdirs as well

2c6e797

one more..

9807bf1

resolve a couple of more path obj to path strs

1e7284d

fix bootstrap testing

dcbea57

fix cli store test

36c65f0

fix test logging to file

b7a1161

fix a few more tests to use pathlib

b234c5e

shutil instead of find_executable

6c1b3a4

ah, str, not path

7087675

dnil marked this pull request as ready for review December 4, 2024 12:32

dnil requested a review from northwestwitch December 4, 2024 12:33

northwestwitch approved these changes Dec 4, 2024

View reviewed changes

dnil and others added 3 commits December 4, 2024 14:31

Use more modern Dockerfile

eabbab9

Update chanjo/testutils.py

3d2abf5

Co-authored-by: Chiara Rasi <rasi.chiara@gmail.com>

main link

9f5b2da

dnil merged commit cb7e319 into main Dec 4, 2024
1 check passed

dnil deleted the migrate_pkg_resources branch December 4, 2024 16:12

This was referenced Dec 10, 2024

Version 4.7.2 - patch release, changelog #267

Merged

Fix action version, bump to 4.7.2 manually #270

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix #261 - pkg_resources migration #262

Fix #261 - pkg_resources migration #262

dnil commented Nov 29, 2024 •

edited

Loading

dnil commented Dec 4, 2024

northwestwitch commented Dec 4, 2024

northwestwitch commented Dec 4, 2024 •

edited

Loading

northwestwitch commented Dec 4, 2024

dnil commented Dec 4, 2024

northwestwitch left a comment

northwestwitch Dec 4, 2024

dnil Dec 4, 2024

northwestwitch Dec 4, 2024

dnil Dec 4, 2024

dnil commented Dec 4, 2024 •

edited

Loading

		@@ -16,14 +19,20 @@

		LOG = logging.getLogger(__name__)

		COMMAND_GROUP_KEY = 'chanjo.subcommands.4'

Fix #261 - pkg_resources migration #262

Fix #261 - pkg_resources migration #262

Conversation

dnil commented Nov 29, 2024 • edited Loading

Description

Fixed

How to prepare for test

How to test

Expected test outcome

Review

This version is a

Implementation Plan

dnil commented Dec 4, 2024

northwestwitch commented Dec 4, 2024

northwestwitch commented Dec 4, 2024 • edited Loading

northwestwitch commented Dec 4, 2024

dnil commented Dec 4, 2024

northwestwitch left a comment

Choose a reason for hiding this comment

northwestwitch Dec 4, 2024

Choose a reason for hiding this comment

dnil Dec 4, 2024

Choose a reason for hiding this comment

northwestwitch Dec 4, 2024

Choose a reason for hiding this comment

dnil Dec 4, 2024

Choose a reason for hiding this comment

dnil commented Dec 4, 2024 • edited Loading

dnil commented Nov 29, 2024 •

edited

Loading

northwestwitch commented Dec 4, 2024 •

edited

Loading

dnil commented Dec 4, 2024 •

edited

Loading