[202311] ./fsroot-broadcom pip3 install --no-build-isolation pygobject error

[yaoting-chang]

Description

Compiling the SONIC 202311 source code in Ubuntu 20.04 system by clean build. There is ModuleNotFoundError: No module named 'mesonpy' error when build SONIC 202311 branch (commit #20250 bba184c) for broadcom platform. The issue is also described in #19602.
How can we solved it?

Steps to reproduce the issue:

1. git clone https://github.com/sonic-net/sonic-buildimage.git
2. sudo modprobe overlay && cd sonic-buildimage
3. git checkout 202311
4. make init
5. make configure PLATFORM=broadcom
6. make target/sonic-broadcom.bin

Describe the results you received:

+ sudo https_proxy= LANG=C chroot ./fsroot-broadcom pip3 install --no-build-isolation pygobject
Collecting pygobject
  Downloading pygobject-3.50.0.tar.gz (1.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 486.4 kB/s eta 0:00:00
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
ERROR: Exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/pip/_internal/cli/base_command.py", line 105, in _run_wrapper
    status = _inner_run()
  File "/usr/local/lib/python3.9/dist-packages/pip/_internal/cli/base_command.py", line 96, in _inner_run
    return self.run(options, args)
  File "/usr/local/lib/python3.9/dist-packages/pip/_internal/cli/req_command.py", line 67, in wrapper
    return func(self, options, args)
  File "/usr/local/lib/python3.9/dist-packages/pip/_internal/commands/install.py", line 379, in run
    requirement_set = resolver.resolve(
  File "/usr/local/lib/python3.9/dist-packages/pip/_internal/resolution/resolvelib/resolver.py", line 95, in resolve
    result = self._result = resolver.resolve(
  File "/usr/local/lib/python3.9/dist-packages/pip/_vendor/resolvelib/resolvers.py", line 546, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/usr/local/lib/python3.9/dist-packages/pip/_vendor/resolvelib/resolvers.py", line 397, in resolve
    self._add_to_criteria(self.state.criteria, r, parent=None)
  File "/usr/local/lib/python3.9/dist-packages/pip/_vendor/resolvelib/resolvers.py", line 173, in _add_to_criteria
    if not criterion.candidates:
  File "/usr/local/lib/python3.9/dist-packages/pip/_vendor/resolvelib/structs.py", line 156, in __bool__
    return bool(self._sequence)
  File "/usr/local/lib/python3.9/dist-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 174, in __bool__
    return any(self)
  File "/usr/local/lib/python3.9/dist-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 162, in <genexpr>
    return (c for c in iterator if id(c) not in self._incompatible_ids)
  File "/usr/local/lib/python3.9/dist-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 53, in _iter_built
    candidate = func()
  File "/usr/local/lib/python3.9/dist-packages/pip/_internal/resolution/resolvelib/factory.py", line 187, in _make_candidate_from_link
    base: Optional[BaseCandidate] = self._make_base_candidate_from_link(
  File "/usr/local/lib/python3.9/dist-packages/pip/_internal/resolution/resolvelib/factory.py", line 233, in _make_base_candidate_from_link
    self._link_candidate_cache[link] = LinkCandidate(
  File "/usr/local/lib/python3.9/dist-packages/pip/_internal/resolution/resolvelib/candidates.py", line 304, in __init__
    super().__init__(
  File "/usr/local/lib/python3.9/dist-packages/pip/_internal/resolution/resolvelib/candidates.py", line 159, in __init__
    self.dist = self._prepare()
  File "/usr/local/lib/python3.9/dist-packages/pip/_internal/resolution/resolvelib/candidates.py", line 236, in _prepare
    dist = self._prepare_distribution()
  File "/usr/local/lib/python3.9/dist-packages/pip/_internal/resolution/resolvelib/candidates.py", line 315, in _prepare_distribution
    return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
  File "/usr/local/lib/python3.9/dist-packages/pip/_internal/operations/prepare.py", line 527, in prepare_linked_requirement
    return self._prepare_linked_requirement(req, parallel_builds)
  File "/usr/local/lib/python3.9/dist-packages/pip/_internal/operations/prepare.py", line 642, in _prepare_linked_requirement
    dist = _get_prepared_distribution(
  File "/usr/local/lib/python3.9/dist-packages/pip/_internal/operations/prepare.py", line 72, in _get_prepared_distribution
    abstract_dist.prepare_distribution_metadata(
  File "/usr/local/lib/python3.9/dist-packages/pip/_internal/distributions/sdist.py", line 69, in prepare_distribution_metadata
    self.req.prepare_metadata()
  File "/usr/local/lib/python3.9/dist-packages/pip/_internal/req/req_install.py", line 575, in prepare_metadata
    self.metadata_directory = generate_metadata(
  File "/usr/local/lib/python3.9/dist-packages/pip/_internal/operations/build/metadata.py", line 35, in generate_metadata
    distinfo_dir = backend.prepare_metadata_for_build_wheel(metadata_dir)
  File "/usr/local/lib/python3.9/dist-packages/pip/_internal/utils/misc.py", line 722, in prepare_metadata_for_build_wheel
    return super().prepare_metadata_for_build_wheel(
  File "/usr/local/lib/python3.9/dist-packages/pip/_vendor/pyproject_hooks/_impl.py", line 186, in prepare_metadata_for_build_wheel
    return self._call_hook('prepare_metadata_for_build_wheel', {
  File "/usr/local/lib/python3.9/dist-packages/pip/_vendor/pyproject_hooks/_impl.py", line 321, in _call_hook
    raise BackendUnavailable(data.get('traceback', ''))
pip._vendor.pyproject_hooks._impl.BackendUnavailable: Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 77, in _build_backend
    obj = import_module(mod_path)
  File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'mesonpy'

+ clean_sys
+ sudo chroot ./fsroot-broadcom umount /sys/fs/cgroup/blkio /sys/fs/cgroup/cpu /sys/fs/cgroup/cpu,cpuacct /sys/fs/cgroup/cpuacct /sys/fs/cgroup/cpuset /sys/fs/cgroup/devices /sys/fs/cgroup/freezer /sys/fs/cgroup/hugetlb /sys/fs/cgroup/memory /sys/fs/cgroup/misc /sys/fs/cgroup/net_cls /sys/fs/cgroup/net_cls,net_prio /sys/fs/cgroup/net_prio /sys/fs/cgroup/perf_event /sys/fs/cgroup/pids /sys/fs/cgroup/rdma /sys/fs/cgroup/systemd /sys/fs/cgroup /sys
umount: /sys/fs/cgroup/cpu: no mount point specified.
umount: /sys/fs/cgroup/cpu,cpuacct: no mount point specified.
umount: /sys/fs/cgroup/cpuacct: no mount point specified.
umount: /sys/fs/cgroup/net_cls: no mount point specified.
umount: /sys/fs/cgroup/net_cls,net_prio: no mount point specified.
umount: /sys/fs/cgroup/net_prio: no mount point specified.
umount: /sys/fs/cgroup/systemd: no mount point specified.
+ true
+ true
+ sudo LANG=C chroot ./fsroot-broadcom umount /proc
+ true
[  FAIL LOG END  ] [ target/sonic-broadcom.bin ]
make: *** [slave.mk:1369: target/sonic-broadcom.bin] Error 1
make[1]: *** [Makefile.work:622: target/sonic-broadcom.bin] Error 2
make[1]: Leaving directory '/media/eric/test/dd-s64c8v2/sonic-buildimage'
make: *** [Makefile:41: target/sonic-broadcom.bin] Error 2

[jhli-cisco]

we are seeing similar failure using 202305 also

[prgeor]

@adyeung anything you can take a look on this? @saiarcot895 fyi if you have some idea..

[adyeung]

@prgeor @saiarcot895 this seems to be a generic issue with 202311 not specific to a target build, #19602 reported the same with platform vs

[yaoting-chang]

We try to fall back to sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT pip3 install pygobject in sonic_debian_extension.j2 and it can build successfully.
files/build_templates/sonic_debian_extension.j2: 48bf9c5