Skip to content

Commit

Permalink
Support require-libpython in the wheel
Browse files Browse the repository at this point in the history
If we're in a user-local install or a venv and we have no libpython SOs linked,
create symlinks and fail if we can't

Release 0.0.6
  • Loading branch information
arcivanov committed Sep 19, 2024
1 parent af7241d commit 0e24af9
Show file tree
Hide file tree
Showing 10 changed files with 327 additions and 71 deletions.
79 changes: 14 additions & 65 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,69 +7,15 @@ on:
branches:
- master
jobs:
build-primary:
runs-on: ${{ matrix.os }}
continue-on-error: false
strategy:
fail-fast: false
matrix:
os:
- ubuntu-latest
- macos-13
python-version:
- '3.12'
- '3.11'
- '3.10'
pip-version:
- '24.2'
- '24.1'
- '24.0'
- '23.3'
- '22.3'
setuptools-version:
- '72.1'
- '71.1'
- '70.3'
- '69.5'
- '68.2'
- '67.8'
- '66.1'
- '65.7'
exclude:
- python-version: '3.12'
setuptools-version: '65.7'
- python-version: '3.12'
pip-version: '22.3'
env:
DEPLOY_PYTHONS: "3.12"
DEPLOY_OSES: "Linux"
DEPLOY_PIPS: "24.2"
DEPLOY_SETUPTOOLS: "72.1"
TWINE_USERNAME: __token__
TWINE_PASSWORD: ${{ secrets.PYPI_TOKEN }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
steps:
- uses: actions/checkout@v4
- shell: bash
run: |
echo "PYB_EXTRA_ARGS=--no-venvs" >> $GITHUB_ENV
echo "SETUPTOOLS_VER=~=${{matrix.setuptools-version}}" >> $GITHUB_ENV
echo "PIP_VER=~=${{matrix.pip-version}}" >> $GITHUB_ENV
- shell: bash
if: |
github.event_name == 'push' &&
contains(env.DEPLOY_OSES, runner.os) &&
contains(env.DEPLOY_PYTHONS, matrix.python-version) &&
contains(env.DEPLOY_PIPS, matrix.pip-version) &&
contains(env.DEPLOY_SETUPTOOLS, matrix.setuptools-version)
run: |
echo "PYB_EXTRA_ARGS=+upload --no-venvs" >> $GITHUB_ENV
- uses: pybuilder/build@master
with:
checkout: false
with-venv: false
python-version: ${{ matrix.python-version }}
pyb-extra-args: ${{ env.PYB_EXTRA_ARGS }}
build-ubuntu-py312:
uses: ./template.yml
with:
os: ubuntu-latest
python-version: '3.12'
deploy: ${{ github.event_name == 'push' }}
deploy-pip: '24.2'
deploy-setuptools: '75.1'
secrets: inherit

build-secondary:
runs-on: ${{ matrix.os }}
Expand All @@ -90,7 +36,10 @@ jobs:
- '23.3'
- '22.3'
setuptools-version:
- '72.1'
- '75.1'
- '74.1'
- '73.0'
- '72.2'
- '71.1'
- '70.3'
- '69.5'
Expand Down Expand Up @@ -131,7 +80,7 @@ jobs:
pip-version:
- '24.2'
setuptools-version:
- '72.1'
- '75.1'
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
steps:
Expand Down
91 changes: 91 additions & 0 deletions .github/workflows/template.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
name: wheel-axle-runtime
on:
workflow_call:
inputs:
os:
required: true
type: string
python-version:
required: true
type: string
deploy:
required: false
type: boolean
default: false
deploy-pip:
required: false
type: string
default: none
deploy-setuptools:
required: false
type: string
exclude:
required: false
type: string
secrets:
PYPI_TOKEN:
required: false

jobs:
build:
runs-on: ${{ matrix.os }}
continue-on-error: false
strategy:
fail-fast: false
matrix:
os:
- ${{ inputs.os }}
python-version:
- ${{ inputs.python-version }}
pip-version:
- '24.2'
- '24.1'
- '24.0'
- '23.3'
- '22.3'
setuptools-version:
- '75.1'
- '74.1'
- '73.0'
- '72.2'
- '71.1'
- '70.3'
- '69.5'
- '68.2'
- '67.8'
- '66.1'
- '65.7'
- '64.0'
- '63.4'
- '62.6'
exclude:
- python-version: '3.12'
setuptools-version: '65.7'
- python-version: '3.12'
pip-version: '22.3'
env:
DEPLOY_PIP: ${{ inputs.deploy-pip }}
DEPLOY_SETUPTOOLS: ${{ inputs.deploy-setuptools }}
TWINE_USERNAME: __token__
TWINE_PASSWORD: ${{ secrets.PYPI_TOKEN }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
steps:
- uses: actions/checkout@v4
- shell: bash
run: |
echo "PYB_EXTRA_ARGS=--no-venvs" >> $GITHUB_ENV
echo "SETUPTOOLS_VER=~=${{matrix.setuptools-version}}" >> $GITHUB_ENV
echo "PIP_VER=~=${{matrix.pip-version}}" >> $GITHUB_ENV
- shell: bash
if: |
inputs.deploy &&
contains(env.DEPLOY_PIP, matrix.pip-version) &&
contains(env.DEPLOY_SETUPTOOLS, matrix.setuptools-version)
run: |
echo "PYB_EXTRA_ARGS=+upload --no-venvs" >> $GITHUB_ENV
- uses: pybuilder/build@master
with:
checkout: false
with-venv: false
python-version: ${{ matrix.python-version }}
pyb-extra-args: ${{ env.PYB_EXTRA_ARGS }}
9 changes: 7 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,14 +50,19 @@ Once the distribution-specific `.pth` is executed by the Python interpreter, the
5. Now that in-process and inter-process race conditions are excluded the post-install work can begin.
6. Registered `installers` are run in sequence. Installers *should be* idempotent. The following installers are
currently implemented:
1. *Symlinks installer* processes `.dist-info/symlinks.txt`, if any.
1. *LibPython installer* checks for the presence of `.dist-info/require-libpython`.
1. The installer determines the location of the installation: venv, user or other.
2. The list of all libpython library files is located from the `sys.base_exec_prefix`.
3. If the installation is either venv or user and the link to the libpython library doesn't exist the symlink
is created.
2. *Symlinks installer* processes `.dist-info/symlinks.txt`, if any.
1. Based on the location of the `.pth` file being executed the current installation `schema` and its paths are
determined. Currently, installation into a virtual environment or user location is supported and tested.
2. For each symlink the target path is resolved and `realpath` is used to determine the final target path.
3. If the symlink path and symlink target path are within one of the permitted schema locations the symlink is
created. Otherwise, an exception is raised and the processing is aborted.
4. After all symlinks are created, the `.dist-info/RECORD` file is updated to reflect the created symlinks.
2. *Axle installer* finalizes the installation. This installer is always executed last.
3. *Axle installer* finalizes the installation. This installer is always executed last.
1. The `.dist-info/RECORD` is updated with `.dist-info/axle.done` file record.
2. `.dist-info/axle.done` is created.
3. `<distribution name and version>.pth` is then removed. If the file cannot be removed it is left in place.
Expand Down
3 changes: 2 additions & 1 deletion build.py
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@
use_plugin("filter_resources")

name = "wheel-axle-runtime"
version = "0.0.6.dev"
version = "0.0.6"

summary = "Axle Runtime is the runtime part of the Python Wheel enhancement library"
authors = [Author("Karellen, Inc.", "supervisor@karellen.co")]
Expand Down Expand Up @@ -100,6 +100,7 @@ def set_properties(project):
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
"Programming Language :: Python :: 3.13",
"Operating System :: MacOS :: MacOS X",
"Operating System :: POSIX",
"Operating System :: POSIX :: Linux",
Expand Down
3 changes: 1 addition & 2 deletions src/integrationtest/python/instrumented_axle_tests.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,6 @@

import pkg_resources
from pip._internal.locations import get_scheme
from pip._internal.utils.virtualenv import virtualenv_no_global


class InstrumentedAxleTest(unittest.TestCase):
Expand Down Expand Up @@ -104,7 +103,7 @@ def test_verify_install(self):

self.assertFalse(exists(dist_info))

@unittest.skipIf(virtualenv_no_global(), "no user site available under virtualenv")
@unittest.skipIf(sys.base_prefix != sys.prefix, "no user site available under virtualenv")
def test_verify_user_install(self):
self.install(self.wheel_file, True)

Expand Down
112 changes: 112 additions & 0 deletions src/integrationtest/python/require_libpython_axle_tests.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,112 @@
# -*- coding: utf-8 -*-
#
# (C) Copyright 2022 Karellen, Inc. (https://www.karellen.co/)
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

import site
import sys
import sysconfig
import unittest
from os.path import join as jp, exists, basename, islink

import pkg_resources
from pip._internal.locations import get_scheme

from instrumented_axle_tests import InstrumentedAxleTest


class RequireLibPythonAxleTest(InstrumentedAxleTest):
def setUp(self) -> None:
super().setUp()

self.wheel_file = jp(self.test_dir, "test_axle_2_libpython-0.0.1-py3-none-any.whl")

def check_libpython_present(self, lib_dir):
enable_shared = sysconfig.get_config_var("PY_ENABLE_SHARED")
if not enable_shared or not int(enable_shared):
return

in_venv = sys.base_exec_prefix != sys.exec_prefix
is_user_site = lib_dir.startswith(site.USER_SITE)
if in_venv or is_user_site:
self.assertTrue(islink(jp(lib_dir, sysconfig.get_config_var("LDLIBRARY"))))
self.assertTrue(islink(jp(lib_dir, sysconfig.get_config_var("INSTSONAME"))))

def test_install_uninstall(self):
self.install(self.wheel_file)
self.uninstall(self.wheel_file)

def test_verify_install(self):
self.install(self.wheel_file)

ws = pkg_resources.WorkingSet()
list(map(ws.add_entry, sys.path))
pkg = ws.by_key["test-axle-2-libpython"]
scheme = get_scheme("test-axle-2-libpython")

prefix = scheme.purelib
pth_file = basename(pkg.egg_info[:-len("dist-info")] + "pth")
pth_path = jp(prefix, pth_file)
dist_info = jp(prefix, basename(pkg.egg_info))
axle_done = jp(dist_info, "axle.done")

self.assertTrue(exists(pth_path))
self.assertFalse(exists(axle_done))

site.addpackage(prefix, pth_file, None)

self.assertFalse(exists(pth_path))
self.assertTrue(exists(axle_done))

self.check_installed_contents(scheme)
self.check_libpython_present(jp(scheme.data, sys.platlibdir))

self.uninstall(self.wheel_file)

self.assertFalse(exists(dist_info))

@unittest.skipIf(sys.base_prefix != sys.prefix, "no user site available under virtualenv")
def test_verify_user_install(self):
self.install(self.wheel_file, True)

ws = pkg_resources.WorkingSet()
list(map(ws.add_entry, sys.path))
ws.add_entry(site.getusersitepackages())
pkg = ws.by_key["test-axle-2-libpython"]
scheme = get_scheme("test-axle-2-libpython", True)

prefix = scheme.purelib
pth_file = basename(pkg.egg_info[:-len("dist-info")] + "pth")
pth_path = jp(prefix, pth_file)
dist_info = jp(prefix, basename(pkg.egg_info))
axle_done = jp(dist_info, "axle.done")

self.assertTrue(exists(pth_path))
self.assertFalse(exists(axle_done))

site.addpackage(prefix, pth_file, None)

self.assertFalse(exists(pth_path))
self.assertTrue(exists(axle_done))

self.check_installed_contents(scheme)

self.uninstall(self.wheel_file)

self.assertFalse(exists(dist_info))


if __name__ == "__main__":
unittest.main()
Binary file not shown.
3 changes: 2 additions & 1 deletion src/main/python/wheel_axle/runtime/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -33,10 +33,11 @@

def _run_installers(dist_info_dir):
# Get metadata
from wheel_axle.runtime._libpython import LibPythonInstaller
from wheel_axle.runtime._symlinks import SymlinksInstaller
from wheel_axle.runtime._axle import AxleFinalizer

installers = [SymlinksInstaller, AxleFinalizer] # AxleFinalizer is always last!
installers = [LibPythonInstaller, SymlinksInstaller, AxleFinalizer] # AxleFinalizer is always last!
for installer in installers:
installer(dist_info_dir).run()

Expand Down
Loading

0 comments on commit 0e24af9

Please sign in to comment.