cibuildwheel failed on pp38-manylinux_x86_64 and pp38-macosx_arm64

[wind23]

Description

My repo and branch is here: https://github.com/wind23/whole_history_rating/tree/dependabot/github_actions/actions-ddff5e9989

Here is my wheels.yml file, or you can find it here: https://github.com/wind23/whole_history_rating/blob/dependabot/github_actions/actions-ddff5e9989/.github/workflows/wheels.yml

name: Wheels

on:
  workflow_dispatch:
  pull_request:
  push:
    branches:
      - master
  release:
    types:
      - published

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

jobs:
  build_sdist:
    name: Build SDist
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Build SDist
        run: pipx run build --sdist

      - name: Check metadata
        run: pipx run twine check dist/*

      - uses: actions/upload-artifact@v4
        with:
          name: cibw-sdist
          path: dist/*.tar.gz

  build_wheels:
    name: Wheels on ${{ matrix.os }}
    runs-on: ${{ matrix.os }}
    strategy:
      fail-fast: false
      matrix:
        os: [ubuntu-latest, windows-latest, macos-13, macos-14]

    steps:
      - uses: actions/checkout@v4

      - uses: pypa/cibuildwheel@v2.21
        env:
          CIBW_ARCHS_MACOS: auto universal2
          SYSTEM_VERSION_COMPAT: 0

      - name: Verify clean directory
        run: git diff --exit-code
        shell: bash

      - name: Upload wheels
        uses: actions/upload-artifact@v4
        with:
          name: cibw-wheels-${{ matrix.os }}
          path: wheelhouse/*.whl

  upload_all:
    name: Upload if release
    needs: [build_wheels, build_sdist]
    runs-on: ubuntu-latest
    if: github.event_name == 'release' && github.event.action == 'published'

    steps:
      - uses: actions/setup-python@v5
        with:
          python-version: "3.x"

      - uses: actions/download-artifact@v4
        with:
          pattern: cibw-*
          path: dist
          merge-multiple: true
      - uses: pypa/gh-action-pypi-publish@release/v1
        with:
          user: __token__
          password: ${{ secrets.pypi_password }}

As you can see, it works on windows-latest, but failed on ubuntu-latest and macos-13. More precisely, it failed on pp38-manylinux_x86_64 and pp38-macosx_x86_64, but successed in the cp3* wheels before.

Also, my same repo used to work a month ago (https://github.com/wind23/whole_history_rating/actions/runs/10270004681), but it fails to work now (https://github.com/wind23/whole_history_rating/actions/runs/10925037181). What would be the problem mostly like?

Here is an example of the failing log:

Building pp38-manylinux_x86_64 wheel
PyPy 3.8 manylinux x86_64

Setting up build environment...
  
      + /opt/python/cp38-cp38/bin/python -c 'import sys, json, os; json.dump(os.environ.copy(), sys.stdout)'
      + which python
      + which pip
                                                              ✓ 0.11s
Building wheel...
  
      + rm -rf /tmp/cibuildwheel/built_wheel
      + mkdir -p /tmp/cibuildwheel/built_wheel
      + python -m pip wheel /project --wheel-dir=/tmp/cibuildwheel/built_wheel --no-deps
  Processing /project
    Installing build dependencies: started
    Installing build dependencies: finished with status 'done'
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'done'
    Preparing metadata (pyproject.toml): started
    Preparing metadata (pyproject.toml): finished with status 'done'
  Building wheels for collected packages: whr
    Building wheel for whr (pyproject.toml): started
    Building wheel for whr (pyproject.toml): finished with status 'error'
    error: subprocess-exited-with-error
    
    × Building wheel for whr (pyproject.toml) did not run successfully.
    │ exit code: 1
    ╰─> [78 lines of output]
        running bdist_wheel
        running build
        running build_py
        creating build/lib.linux-x86_64-pypy38/whr
        copying whr/evaluate.py -> build/lib.linux-x86_64-pypy38/whr
        copying whr/__init__.py -> build/lib.linux-x86_64-pypy38/whr
        copying whr/base.py -> build/lib.linux-x86_64-pypy38/whr
        running build_ext
        g++ -pthread -DNDEBUG -O2 -fPIC -I/opt/python/pp38-pypy38_pp73/include/pypy3.8 -c flagcheck.cpp -o flagcheck.o -std=c++17
        building 'whr_core' extension
        creating build/temp.linux-x86_64-pypy38/src
        g++ -pthread -DNDEBUG -O2 -fPIC -DVERSION_INFO=2.0.3 -I/tmp/pip-build-env-v81ucbg4/overlay/lib/pypy3.8/site-packages/pybind11/include -I/opt/python/pp38-pypy38_pp73/include/pypy3.8 -c src/base.cc -o build/temp.linux-x86_64-pypy38/src/base.o -std=c++17 -fvisibility=hidden -g0
        g++ -pthread -DNDEBUG -O2 -fPIC -DVERSION_INFO=2.0.3 -I/tmp/pip-build-env-v81ucbg4/overlay/lib/pypy3.8/site-packages/pybind11/include -I/opt/python/pp38-pypy38_pp73/include/pypy3.8 -c src/evaluate.cc -o build/temp.linux-x86_64-pypy38/src/evaluate.o -std=c++17 -fvisibility=hidden -g0
        g++ -pthread -DNDEBUG -O2 -fPIC -DVERSION_INFO=2.0.3 -I/tmp/pip-build-env-v81ucbg4/overlay/lib/pypy3.8/site-packages/pybind11/include -I/opt/python/pp38-pypy38_pp73/include/pypy3.8 -c src/game.cc -o build/temp.linux-x86_64-pypy38/src/game.o -std=c++17 -fvisibility=hidden -g0
        g++ -pthread -DNDEBUG -O2 -fPIC -DVERSION_INFO=2.0.3 -I/tmp/pip-build-env-v81ucbg4/overlay/lib/pypy3.8/site-packages/pybind11/include -I/opt/python/pp38-pypy38_pp73/include/pypy3.8 -c src/player.cc -o build/temp.linux-x86_64-pypy38/src/player.o -std=c++17 -fvisibility=hidden -g0
        g++ -pthread -DNDEBUG -O2 -fPIC -DVERSION_INFO=2.0.3 -I/tmp/pip-build-env-v81ucbg4/overlay/lib/pypy3.8/site-packages/pybind11/include -I/opt/python/pp38-pypy38_pp73/include/pypy3.8 -c src/player_day.cc -o build/temp.linux-x86_64-pypy38/src/player_day.o -std=c++17 -fvisibility=hidden -g0
        g++ -pthread -DNDEBUG -O2 -fPIC -DVERSION_INFO=2.0.3 -I/tmp/pip-build-env-v81ucbg4/overlay/lib/pypy3.8/site-packages/pybind11/include -I/opt/python/pp38-pypy38_pp73/include/pypy3.8 -c src/pybind11.cc -o build/temp.linux-x86_64-pypy38/src/pybind11.o -std=c++17 -fvisibility=hidden -g0
        Traceback (most recent call last):
          File "/opt/python/pp38-pypy38_pp73/lib/pypy3.8/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
            main()
          File "/opt/python/pp38-pypy38_pp73/lib/pypy3.8/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
            json_out['return_val'] = hook(**hook_input['kwargs'])
          File "/opt/python/pp38-pypy38_pp73/lib/pypy3.8/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 251, in build_wheel
            return _build_backend().build_wheel(wheel_directory, config_settings,
          File "/tmp/pip-build-env-v81ucbg4/overlay/lib/pypy3.8/site-packages/setuptools/build_meta.py", line 421, in build_wheel
            return self._build_with_temp_dir(
          File "/tmp/pip-build-env-v81ucbg4/overlay/lib/pypy3.8/site-packages/setuptools/build_meta.py", line 403, in _build_with_temp_dir
            self.run_setup()
          File "/tmp/pip-build-env-v81ucbg4/overlay/lib/pypy3.8/site-packages/setuptools/build_meta.py", line 318, in run_setup
            exec(code, locals())
          File "<string>", line 20, in <module>
          File "/tmp/pip-build-env-v81ucbg4/overlay/lib/pypy3.8/site-packages/setuptools/__init__.py", line 117, in setup
            return distutils.core.setup(**attrs)
          File "/tmp/pip-build-env-v81ucbg4/overlay/lib/pypy3.8/site-packages/setuptools/_distutils/core.py", line 183, in setup
            return run_commands(dist)
          File "/tmp/pip-build-env-v81ucbg4/overlay/lib/pypy3.8/site-packages/setuptools/_distutils/core.py", line 199, in run_commands
            dist.run_commands()
          File "/tmp/pip-build-env-v81ucbg4/overlay/lib/pypy3.8/site-packages/setuptools/_distutils/dist.py", line 954, in run_commands
            self.run_command(cmd)
          File "/tmp/pip-build-env-v81ucbg4/overlay/lib/pypy3.8/site-packages/setuptools/dist.py", line 950, in run_command
            super().run_command(command)
          File "/tmp/pip-build-env-v81ucbg4/overlay/lib/pypy3.8/site-packages/setuptools/_distutils/dist.py", line 973, in run_command
            cmd_obj.run()
          File "/tmp/pip-build-env-v81ucbg4/overlay/lib/pypy3.8/site-packages/setuptools/command/bdist_wheel.py", line 398, in run
            self.run_command("build")
          File "/tmp/pip-build-env-v81ucbg4/overlay/lib/pypy3.8/site-packages/setuptools/_distutils/cmd.py", line 316, in run_command
            self.distribution.run_command(command)
          File "/tmp/pip-build-env-v81ucbg4/overlay/lib/pypy3.8/site-packages/setuptools/dist.py", line 950, in run_command
            super().run_command(command)
          File "/tmp/pip-build-env-v81ucbg4/overlay/lib/pypy3.8/site-packages/setuptools/_distutils/dist.py", line 973, in run_command
            cmd_obj.run()
          File "/tmp/pip-build-env-v81ucbg4/overlay/lib/pypy3.8/site-packages/setuptools/_distutils/command/build.py", line 135, in run
            self.run_command(cmd_name)
          File "/tmp/pip-build-env-v81ucbg4/overlay/lib/pypy3.8/site-packages/setuptools/_distutils/cmd.py", line 316, in run_command
            self.distribution.run_command(command)
          File "/tmp/pip-build-env-v81ucbg4/overlay/lib/pypy3.8/site-packages/setuptools/dist.py", line 950, in run_command
            super().run_command(command)
          File "/tmp/pip-build-env-v81ucbg4/overlay/lib/pypy3.8/site-packages/setuptools/_distutils/dist.py", line 973, in run_command
            cmd_obj.run()
          File "/tmp/pip-build-env-v81ucbg4/overlay/lib/pypy3.8/site-packages/setuptools/command/build_ext.py", line 98, in run
            _build_ext.run(self)
          File "/tmp/pip-build-env-v81ucbg4/overlay/lib/pypy3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 359, in run
            self.build_extensions()
          File "/tmp/pip-build-env-v81ucbg4/overlay/lib/pypy3.8/site-packages/pybind11/setup_helpers.py", line 287, in build_extensions
            super().build_extensions()
          File "/tmp/pip-build-env-v81ucbg4/overlay/lib/pypy3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 476, in build_extensions
            self._build_extensions_serial()
          File "/tmp/pip-build-env-v81ucbg4/overlay/lib/pypy3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 502, in _build_extensions_serial
            self.build_extension(ext)
          File "/tmp/pip-build-env-v81ucbg4/overlay/lib/pypy3.8/site-packages/setuptools/command/build_ext.py", line 263, in build_extension
            _build_ext.build_extension(self, ext)
          File "/tmp/pip-build-env-v81ucbg4/overlay/lib/pypy3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 581, in build_extension
            self.compiler.link_shared_object(
          File "/tmp/pip-build-env-v81ucbg4/overlay/lib/pypy3.8/site-packages/setuptools/_distutils/ccompiler.py", line 758, in link_shared_object
            self.link(
          File "/tmp/pip-build-env-v81ucbg4/overlay/lib/pypy3.8/site-packages/setuptools/_distutils/unixccompiler.py", line 269, in link
            self.linker_exe
        TypeError: 'NoneType' object is not subscriptable (key slice(None, None, None))
        [end of output]
    
    note: This error originates from a subprocess, and is likely not a problem with pip.
    ERROR: Failed building wheel for whr
  Failed to build whr
  ERROR: Failed to build one or more wheels
                                                             ✕ 20.46s
Error: Command ['python', '-m', 'pip', 'wheel', PurePosixPath('/project'), '--wheel-dir=/tmp/cibuildwheel/built_wheel', '--no-deps'] failed with code 1. 

Error: Process completed with exit code 1.

My repo is written according to the python example of pybind: https://github.com/pybind/python_example/
As you can see, the wheels of their repo is also failing: https://github.com/pybind/python_example/actions/runs/10893406296/job/30228259053

Build log

https://github.com/wind23/whole_history_rating/actions/runs/10925037181/job/30325566939

CI config

https://github.com/wind23/whole_history_rating/blob/dependabot/github_actions/actions-ddff5e9989/.github/workflows/wheels.yml

[Czaki]

It is the same bug as #1983. It is pypy and recent setuptools incompatibility.

You need either to skip pyp38 or pin lower setuptools version for this not maintained pypy version.

[wind23]

It is the same bug as #1983. It is pypy and recent setuptools incompatibility.

You need either to skip pyp38 or pin lower setuptools version for this not maintained pypy version.

Thank you. I skipped pypy in the pipeline and it works now.

BTW, in #1983, it mentioned that pypy310 should work. However, in my project, pypy310 failed to work on Mac OS, although it is working on Windows and Ubuntu:
https://github.com/wind23/whole_history_rating/actions/runs/10925866795/job/30328408987

Building pp310-macosx_arm64 wheel
PyPy 3.10 macOS arm64 - Apple Silicon

Installing Python pp310...
                                                              ✓ 2.30s
Setting up build environment...
                                                              ✓ 2.52s
Installing build tools...
                                                              ✓ 1.36s
Building wheel...
  
  + python -m pip wheel /Users/runner/work/whole_history_rating/whole_history_rating --wheel-dir=/private/var/folders/n3/x0zngmn15fzg2jfkglj4ys4w0000gn/T/cibw-run-4dghxnj3/pp310-macosx_arm64/built_wheel --no-deps
  Processing /Users/runner/work/whole_history_rating/whole_history_rating
    Installing build dependencies: started
    Installing build dependencies: finished with status 'done'
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'done'
    Preparing metadata (pyproject.toml): started
    Preparing metadata (pyproject.toml): finished with status 'done'
  Building wheels for collected packages: whr
    Building wheel for whr (pyproject.toml): started
    Building wheel for whr (pyproject.toml): finished with status 'error'
    error: subprocess-exited-with-error
    
    × Building wheel for whr (pyproject.toml) did not run successfully.
    │ exit code: 1
    ╰─> [21 lines of output]
        running bdist_wheel
        running build
        running build_py
        creating build/lib.macosx-11.0-arm64-pypy310/whr
        copying whr/__init__.py -> build/lib.macosx-11.0-arm64-pypy310/whr
        copying whr/evaluate.py -> build/lib.macosx-11.0-arm64-pypy310/whr
        copying whr/base.py -> build/lib.macosx-11.0-arm64-pypy310/whr
        running build_ext
        g++ -pthread -DNDEBUG -O2 -fPIC -arch arm64 -I/private/var/folders/n3/x0zngmn15fzg2jfkglj4ys4w0000gn/T/cibw-run-4dghxnj3/pp310-macosx_arm64/build/venv/include -I/Users/runner/Library/Caches/cibuildwheel/pypy3.10-v7.3.17-macos_arm64/include/pypy3.10 -c flagcheck.cpp -o flagcheck.o -std=c++17
        building 'whr_core' extension
        creating build/temp.macosx-11.0-arm64-pypy310/src
        g++ -pthread -DNDEBUG -O2 -fPIC -arch arm64 -DVERSION_INFO=2.0.4 -I/private/var/folders/n3/x0zngmn15fzg2jfkglj4ys4w0000gn/T/pip-build-env-ev9sgrmb/overlay/lib/pypy3.10/site-packages/pybind11/include -I/private/var/folders/n3/x0zngmn15fzg2jfkglj4ys4w0000gn/T/cibw-run-4dghxnj3/pp310-macosx_arm64/build/venv/include -I/Users/runner/Library/Caches/cibuildwheel/pypy3.10-v7.3.17-macos_arm64/include/pypy3.10 -c src/base.cc -o build/temp.macosx-11.0-arm64-pypy310/src/base.o -std=c++17 -fvisibility=hidden -g0
        g++ -pthread -DNDEBUG -O2 -fPIC -arch arm64 -DVERSION_INFO=2.0.4 -I/private/var/folders/n3/x0zngmn15fzg2jfkglj4ys4w0000gn/T/pip-build-env-ev9sgrmb/overlay/lib/pypy3.10/site-packages/pybind11/include -I/private/var/folders/n3/x0zngmn15fzg2jfkglj4ys4w0000gn/T/cibw-run-4dghxnj3/pp310-macosx_arm64/build/venv/include -I/Users/runner/Library/Caches/cibuildwheel/pypy3.10-v7.3.17-macos_arm64/include/pypy3.10 -c src/evaluate.cc -o build/temp.macosx-11.0-arm64-pypy310/src/evaluate.o -std=c++17 -fvisibility=hidden -g0
        g++ -pthread -DNDEBUG -O2 -fPIC -arch arm64 -DVERSION_INFO=2.0.4 -I/private/var/folders/n3/x0zngmn15fzg2jfkglj4ys4w0000gn/T/pip-build-env-ev9sgrmb/overlay/lib/pypy3.10/site-packages/pybind11/include -I/private/var/folders/n3/x0zngmn15fzg2jfkglj4ys4w0000gn/T/cibw-run-4dghxnj3/pp310-macosx_arm64/build/venv/include -I/Users/runner/Library/Caches/cibuildwheel/pypy3.10-v7.3.17-macos_arm64/include/pypy3.10 -c src/game.cc -o build/temp.macosx-11.0-arm64-pypy310/src/game.o -std=c++17 -fvisibility=hidden -g0
        g++ -pthread -DNDEBUG -O2 -fPIC -arch arm64 -DVERSION_INFO=2.0.4 -I/private/var/folders/n3/x0zngmn15fzg2jfkglj4ys4w0000gn/T/pip-build-env-ev9sgrmb/overlay/lib/pypy3.10/site-packages/pybind11/include -I/private/var/folders/n3/x0zngmn15fzg2jfkglj4ys4w0000gn/T/cibw-run-4dghxnj3/pp310-macosx_arm64/build/venv/include -I/Users/runner/Library/Caches/cibuildwheel/pypy3.10-v7.3.17-macos_arm64/include/pypy3.10 -c src/player.cc -o build/temp.macosx-11.0-arm64-pypy310/src/player.o -std=c++17 -fvisibility=hidden -g0
        g++ -pthread -DNDEBUG -O2 -fPIC -arch arm64 -DVERSION_INFO=2.0.4 -I/private/var/folders/n3/x0zngmn15fzg2jfkglj4ys4w0000gn/T/pip-build-env-ev9sgrmb/overlay/lib/pypy3.10/site-packages/pybind11/include -I/private/var/folders/n3/x0zngmn15fzg2jfkglj4ys4w0000gn/T/cibw-run-4dghxnj3/pp310-macosx_arm64/build/venv/include -I/Users/runner/Library/Caches/cibuildwheel/pypy3.10-v7.3.17-macos_arm64/include/pypy3.10 -c src/player_day.cc -o build/temp.macosx-11.0-arm64-pypy310/src/player_day.o -std=c++17 -fvisibility=hidden -g0
        g++ -pthread -DNDEBUG -O2 -fPIC -arch arm64 -DVERSION_INFO=2.0.4 -I/private/var/folders/n3/x0zngmn15fzg2jfkglj4ys4w0000gn/T/pip-build-env-ev9sgrmb/overlay/lib/pypy3.10/site-packages/pybind11/include -I/private/var/folders/n3/x0zngmn15fzg2jfkglj4ys4w0000gn/T/cibw-run-4dghxnj3/pp310-macosx_arm64/build/venv/include -I/Users/runner/Library/Caches/cibuildwheel/pypy3.10-v7.3.17-macos_arm64/include/pypy3.10 -c src/pybind11.cc -o build/temp.macosx-11.0-arm64-pypy310/src/pybind11.o -std=c++17 -fvisibility=hidden -g0
        g++ -pthread -DNDEBUG -O2 -shared -Wl,-O1 -Wl,-Bsymbolic-functions -arch arm64 build/temp.macosx-11.0-arm64-pypy310/src/base.o build/temp.macosx-11.0-arm64-pypy310/src/evaluate.o build/temp.macosx-11.0-arm64-pypy310/src/game.o build/temp.macosx-11.0-arm64-pypy310/src/player.o build/temp.macosx-11.0-arm64-pypy310/src/player_day.o build/temp.macosx-11.0-arm64-pypy310/src/pybind11.o -o build/lib.macosx-11.0-arm64-pypy310/whr_core.pypy310-pp73-darwin.so
        ld: unknown options: -Bsymbolic-functions
        clang: error: linker command failed with exit code 1 (use -v to see invocation)
        error: command '/usr/bin/g++' failed with exit code 1
        [end of output]
    
    note: This error originates from a subprocess, and is likely not a problem with pip.
    ERROR: Failed building wheel for whr
  Failed to build whr
  ERROR: Failed to build one or more wheels
                                                             ✕ 11.37s
Error: Command ['python', '-m', 'pip', 'wheel', '/Users/runner/work/whole_history_rating/whole_history_rating', '--wheel-dir=/private/var/folders/n3/x0zngmn15fzg2jfkglj4ys4w0000gn/T/cibw-run-4dghxnj3/pp310-macosx_arm64/built_wheel', '--no-deps'] failed with code 1. 

Error: Process completed with exit code 1.

[QuLogic]

The setuptools bug is pypa/distutils#283; you can pin setuptools<72.2.0 on PyPy builds to get it working.