Python 3rdParty Dependency not Resolving Correctly

[philnugent]

I am trying to add azure-appconfiguration-provider library as a third party dependency to my project. When I execute the run goal to run the application, I have no problems. When I run package to build a pex package, I get the error ModuleNotFoundError: No module named 'azure.appconfiguration.provider'. The library includes a transitive dependency named azure-appconfiguration - my guess is that this transitive dependency is keeping the primary dependency from resolving. I'm new to Pants and Pex, so I'm not sure where this could be happening. I've tried explicitly mapping the module to ensure it is included. Is there any way to explicitly ensure a dependency is included in the Pex package? I'm at a loss for how to resolve this. My relevant configuration is below and am using Pants version 2.21.1. If anyone can shed some light on what could be going on or how to resolve this, I would be eternally grateful.

src/3rdparty/requirements.txt

...
azure-appconfiguration-provider==1.3.0
azure-appconfiguration==1.7.1
...

src/3rdparty/BUILD

python_requirements(
    module_mapping={
        "azure-appconfiguration-provider": ["azure.appconfiguration.provider"],
    },
)

src/application/BUILD

python_sources(
    name="application",
    sources=["*.py"],
)

pex_binary(
    name="local-binary",
    entry_point="application.py",
    dependencies=[
        "src/3rdparty#azure-appconfiguration-provider",
    ],
)

[philnugent]

Looks like it is an issue with Pex. I created a simple project and packaged with Pex I get the same error. However, if anyone here knows how to resolve this issue in Pex, I would be grateful. This issue is likely being caused by a module name being confused as a transitive dependency's submodule, e.g. dependency foo.bar.baz has transitive dependency foo.bar

[benjyw]

What is your value for this option ? Often when pex has trouble with this sort of thing it's because of funky dists that aren't amenable to symlinking. You can experiment with this in standalone pex with the --venv-site-packages-copies flag.

[benjyw]

(subpackages depending on parent packages would be a good example of "funky"...)

[philnugent]

Thanks @benjyw! That completely resolved it! With a little finagling I figured out the right combination of settings. I had the default set with venv_use_symlinks = false.

In my pex test, I i added the switches --venv --venv-site-packages-copies which allowed me to build and execute.

Adding the config below to pant.toml allowed me to package, but not run:

[pex]
venv_use_symlinks = true

I'm assuming there is a way to define execution_mode globally in Pants for pex subsystem. If I had set that to venv I assume that would have worked as well.

Adding the pex switches to the target in my build file allowed me to package and run the pex binary via Pants:

pex_binary(
    name="local-binary",
    entry_point="application.py",
    execution_mode="venv",
    venv_site_packages_copies=True,
)

[benjyw]

Glad that worked! Looks like there is not a global way to set the execution_mode, alas. In the new python backend I want to unify BUILD files and config (by which I mostly mean getting rid of BUILD files in favor of local config files) so that there is one obvious way to set any knob globally or locally.