-
Notifications
You must be signed in to change notification settings - Fork 540
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
refactor!(bzlmod): introduce pypi.install extension #2278
Changes from all commits
fd47903
f7e65fe
4f6f24f
9892c86
33d5ae4
1e7946d
93f8d4c
89bed46
4d14acb
4800a0a
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
|
@@ -13,10 +13,10 @@ Using PyPI packages (aka "pip install") involves two main steps. | |||||
|
||||||
### Using bzlmod | ||||||
|
||||||
To add pip dependencies to your `MODULE.bazel` file, use the `pip.parse` | ||||||
extension, and call it to create the central external repo and individual wheel | ||||||
external repos. Include in the `MODULE.bazel` the toolchain extension as shown | ||||||
in the first bzlmod example above. | ||||||
To add pip dependencies to your `MODULE.bazel` file, use the {bzl:obj}`pip.parse` or | ||||||
{bzl:obj}`pypi.install` extension, and call it to create the central external | ||||||
repo and individual wheel external repos. Include in the `MODULE.bazel` the | ||||||
toolchain extension as shown in the first bzlmod example above. | ||||||
|
||||||
```starlark | ||||||
pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip") | ||||||
|
@@ -27,6 +27,7 @@ pip.parse( | |||||
) | ||||||
use_repo(pip, "my_deps") | ||||||
``` | ||||||
|
||||||
For more documentation, including how the rules can update/create a requirements | ||||||
file, see the bzlmod examples under the {gh-path}`examples` folder or the documentation | ||||||
for the {obj}`@rules_python//python/extensions:pip.bzl` extension. | ||||||
|
@@ -44,6 +45,78 @@ hermetic host python interpreter on this platform. Linux and OSX users should se | |||||
difference. | ||||||
``` | ||||||
|
||||||
(bazel-downloader)= | ||||||
#### Using `pypi.install` extension | ||||||
|
||||||
:::{warning} | ||||||
The APIs here have been well tested but may still change in backwards | ||||||
incompatible ways if major design flaws are found. | ||||||
::: | ||||||
|
||||||
`pypi.install` provides almost a drop-in replacement for the `pip.parse` extension: | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Have this link to the api doc |
||||||
```starlark | ||||||
pypi = use_extension("@rules_python//python/extensions:pypi.bzl", "pypi") | ||||||
pypi.install( | ||||||
hub_name = "my_deps", | ||||||
python_version = "3.11", | ||||||
requirements_lock = "//:requirements_lock_3_11.txt", | ||||||
) | ||||||
use_repo(pypi, "my_deps") | ||||||
``` | ||||||
|
||||||
The `bzlmod` `pypi.install` call supports pulling information from `PyPI` (or a | ||||||
compatible mirror) and it will ensure that the [bazel | ||||||
downloader][bazel_downloader] is used for downloading the wheels or the sdists | ||||||
in the requirements file. This allows the users to use the [credential | ||||||
helper](#credential-helper) to authenticate with the mirror and it also ensures | ||||||
that the distribution downloads are cached. | ||||||
It also avoids using `pip` altogether and results in much faster dependency | ||||||
fetching. | ||||||
|
||||||
See the {gh-path}`examples/bzlmod/MODULE.bazel` example for more info. | ||||||
|
||||||
:::{warning} | ||||||
For now there is a limitation that the lock | ||||||
files consumed by the `pypi.install` extension must have hashes, similarly to | ||||||
all our examples. Hence, the `requirements.txt` files produced by `pip lock` or | ||||||
by `uv pip compile` but without generating hashes will fail. | ||||||
::: | ||||||
|
||||||
Note, when using this feature during the `pip` extension evaluation you will see the accessed indexes similar to below: | ||||||
```console | ||||||
Loading: 0 packages loaded | ||||||
currently loading: docs/ | ||||||
Fetching module extension pip in @@//python/extensions:pypi.bzl; starting | ||||||
Fetching https://pypi.org/simple/twine/ | ||||||
``` | ||||||
|
||||||
This does not mean that `rules_python` is fetching the wheels eagerly, but it | ||||||
rather means that it is calling the PyPI server to get the Simple API response | ||||||
to get the list of all available source and wheel distributions. Once it has | ||||||
got all of the available distributions, it will select the right ones depending | ||||||
on the `sha256` values in your `requirements_lock.txt` file. The compatible | ||||||
distribution URLs will be then written to the `MODULE.bazel.lock` file. Currently | ||||||
users wishing to use the lock file with `rules_python` with this feature have | ||||||
to set an environment variable `RULES_PYTHON_OS_ARCH_LOCK_FILE=0` which will | ||||||
become default in the next release. | ||||||
|
||||||
Fetching the distribution information from the PyPI allows `rules_python` to | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
know which `whl` should be used on which target platform and it will determine | ||||||
that by parsing the `whl` filename based on [PEP600], [PEP656] standards. This | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
allows the user to configure the behaviour by using the following publicly | ||||||
rickeylev marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
available flags: | ||||||
* {obj}`--@rules_python//python/config_settings:py_linux_libc` for selecting the Linux libc variant. | ||||||
* {obj}`--@rules_python//python/config_settings:pip_whl` for selecting `whl` distribution preference. | ||||||
* {obj}`--@rules_python//python/config_settings:pip_whl_osx_arch` for selecting MacOS wheel preference. | ||||||
* {obj}`--@rules_python//python/config_settings:pip_whl_glibc_version` for selecting the GLIBC version compatibility. | ||||||
* {obj}`--@rules_python//python/config_settings:pip_whl_muslc_version` for selecting the musl version compatibility. | ||||||
* {obj}`--@rules_python//python/config_settings:pip_whl_osx_version` for selecting MacOS version compatibility. | ||||||
|
||||||
[bazel_downloader]: https://bazel.build/rules/lib/builtins/repository_ctx#download | ||||||
[pep600]: https://peps.python.org/pep-0600/ | ||||||
[pep656]: https://peps.python.org/pep-0656/ | ||||||
|
||||||
|
||||||
### Using a WORKSPACE file | ||||||
|
||||||
To add pip dependencies to your `WORKSPACE`, load the `pip_parse` function and | ||||||
|
@@ -134,6 +207,13 @@ re-executed to pick up a non-hermetic change to your environment (e.g., updating | |||||
your system `python` interpreter), you can force it to re-execute by running | ||||||
`bazel sync --only [pip_parse name]`. | ||||||
|
||||||
:::{note} | ||||||
The same is not necessarily true for the `pypi.install` bazel extension as it is | ||||||
not relying on the built in interpreter environment information and yields the same | ||||||
set of dependencies irrespective of the host machine and the python interpreter used | ||||||
to pull the Python dependencies. | ||||||
::: | ||||||
|
||||||
{#using-third-party-packages} | ||||||
## Using third party packages as dependencies | ||||||
|
||||||
|
@@ -306,55 +386,6 @@ leg of the dependency manually. For instance by making | |||||
`apache-airflow-providers-postgres` not explicitly depend on `apache-airflow` or | ||||||
perhaps `apache-airflow-providers-common-sql`. | ||||||
|
||||||
|
||||||
(bazel-downloader)= | ||||||
### Bazel downloader and multi-platform wheel hub repository. | ||||||
|
||||||
The `bzlmod` `pip.parse` call supports pulling information from `PyPI` (or a | ||||||
compatible mirror) and it will ensure that the [bazel | ||||||
downloader][bazel_downloader] is used for downloading the wheels. This allows | ||||||
the users to use the [credential helper](#credential-helper) to authenticate | ||||||
with the mirror and it also ensures that the distribution downloads are cached. | ||||||
It also avoids using `pip` altogether and results in much faster dependency | ||||||
fetching. | ||||||
|
||||||
This can be enabled by `experimental_index_url` and related flags as shown in | ||||||
the {gh-path}`examples/bzlmod/MODULE.bazel` example. | ||||||
|
||||||
When using this feature during the `pip` extension evaluation you will see the accessed indexes similar to below: | ||||||
```console | ||||||
Loading: 0 packages loaded | ||||||
currently loading: docs/ | ||||||
Fetching module extension pip in @@//python/extensions:pip.bzl; starting | ||||||
Fetching https://pypi.org/simple/twine/ | ||||||
``` | ||||||
|
||||||
This does not mean that `rules_python` is fetching the wheels eagerly, but it | ||||||
rather means that it is calling the PyPI server to get the Simple API response | ||||||
to get the list of all available source and wheel distributions. Once it has | ||||||
got all of the available distributions, it will select the right ones depending | ||||||
on the `sha256` values in your `requirements_lock.txt` file. The compatible | ||||||
distribution URLs will be then written to the `MODULE.bazel.lock` file. Currently | ||||||
users wishing to use the lock file with `rules_python` with this feature have | ||||||
to set an environment variable `RULES_PYTHON_OS_ARCH_LOCK_FILE=0` which will | ||||||
become default in the next release. | ||||||
|
||||||
Fetching the distribution information from the PyPI allows `rules_python` to | ||||||
know which `whl` should be used on which target platform and it will determine | ||||||
that by parsing the `whl` filename based on [PEP600], [PEP656] standards. This | ||||||
allows the user to configure the behaviour by using the following publicly | ||||||
available flags: | ||||||
* {obj}`--@rules_python//python/config_settings:py_linux_libc` for selecting the Linux libc variant. | ||||||
* {obj}`--@rules_python//python/config_settings:pip_whl` for selecting `whl` distribution preference. | ||||||
* {obj}`--@rules_python//python/config_settings:pip_whl_osx_arch` for selecting MacOS wheel preference. | ||||||
* {obj}`--@rules_python//python/config_settings:pip_whl_glibc_version` for selecting the GLIBC version compatibility. | ||||||
* {obj}`--@rules_python//python/config_settings:pip_whl_muslc_version` for selecting the musl version compatibility. | ||||||
* {obj}`--@rules_python//python/config_settings:pip_whl_osx_version` for selecting MacOS version compatibility. | ||||||
|
||||||
[bazel_downloader]: https://bazel.build/rules/lib/builtins/repository_ctx#download | ||||||
[pep600]: https://peps.python.org/pep-0600/ | ||||||
[pep656]: https://peps.python.org/pep-0656/ | ||||||
|
||||||
(credential-helper)= | ||||||
### Credential Helper | ||||||
|
||||||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: move the breaking entries to the top of the changed section