Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Upgrades the dependencies to latest versions #92

Closed
wants to merge 4 commits into from
Closed

Upgrades the dependencies to latest versions #92

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
043eda6
Ugrades the dependencies to latest versions
TrilokGeer Aug 30, 2024
4f480df
Updates ansible collections
TrilokGeer Sep 23, 2024
3e27029
Updates python version to 3.12, community.docker to latest and pipfil…
TrilokGeer Sep 24, 2024
62f4f77
Updates workflow test-ansible and pipfile.lock
TrilokGeer Oct 12, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/test-ansible.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,10 +29,10 @@ jobs:
- run: sudo rm -rf /usr/local/bin/kustomize
- uses: actions/setup-python@v4
with:
python-version: '3.9'
python-version: '3.12'
- name: Run test e2e ansible molecule
run: |
env
pip3 install --user --upgrade setuptools pip
pip3 install --user ansible-core~=2.15.0
pip3 install --user ansible-core~=2.17.4
make test-e2e-ansible-molecule
2 changes: 1 addition & 1 deletion hack/generate/samples/ansible/testdata/inventory/hosts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,4 @@
127.0.0.1 ansible_connection=local

[all:vars]
ansible_python_interpreter=/usr/bin/python3
ansible_python_interpreter=/usr/bin/python3.12
35 changes: 19 additions & 16 deletions images/ansible-operator/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
# It is built with dependencies that take a while to download, thus speeding
# up ansible deploy jobs.

FROM registry.access.redhat.com/ubi8/ubi:8.9-1107 AS basebuilder
FROM registry.access.redhat.com/ubi9/ubi:9.4-1214 AS basebuilder
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we are upgrading the ubi version as well as the python version, I think the pipfile.Dockerfile should also be updated with the similar changes here:

ansible-operator-plugins/images/ansible-operator/pipfile.Dockerfile

Lines 1 to 22 in 42b5d80

FROM registry.access.redhat.com/ubi8/ubi:8.9-1107 AS basebuilder
# Install Rust so that we can ensure backwards compatibility with installing/building the cryptography wheel across all platforms
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
ENV PATH="/root/.cargo/bin:${PATH}"
RUN rustc --version
# Copy python dependencies (including ansible) to be installed using Pipenv
COPY ./Pipfile ./
# Instruct pip(env) not to keep a cache of installed packages,
# to install into the global site-packages and
# to clear the pipenv cache as well
ENV PIP_NO_CACHE_DIR=1 \
PIPENV_SYSTEM=1 \
PIPENV_CLEAR=1
# Ensure fresh metadata rather than cached metadata, install system and pip python deps,
# and remove those not needed at runtime.
RUN set -e && yum clean all && rm -rf /var/cache/yum/* \
&& yum update -y \
&& yum install -y libffi-devel openssl-devel python39-devel gcc python39-pip python39-setuptools \
&& pip3 install --upgrade pip~=23.3.2 \
&& pip3 install pipenv==2023.11.15 \

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah!, missed to check-in this file.


# Install Rust so that we can ensure backwards compatibility with installing/building the cryptography wheel across all platforms
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
Expand All @@ -11,20 +11,22 @@ RUN rustc --version

# Copy python dependencies (including ansible) to be installed using Pipenv
COPY images/ansible-operator/Pipfile* ./

# Instruct pip(env) not to keep a cache of installed packages,
# to install into the global site-packages and
# to clear the pipenv cache as well
ENV PIP_NO_CACHE_DIR=1 \
PIPENV_SYSTEM=1 \
PIPENV_CLEAR=1
ENV PIP_NO_CACHE_DIR=1

# Ensure fresh metadata rather than cached metadata, install system and pip python deps,
# and remove those not needed at runtime.
RUN set -e && yum clean all && rm -rf /var/cache/yum/* \
&& yum update -y \
&& yum install -y libffi-devel openssl-devel python39-devel gcc python39-pip python39-setuptools \
&& pip3 install --upgrade pip~=23.3.2 \
&& pip3 install pipenv==2023.11.15 \
&& pipenv install --deploy \
Copy link
Contributor Author

@TrilokGeer TrilokGeer Sep 15, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pipenv usage seems to contradict by installing packages to global path and also, blocks usage of different version when used with requires version. As the container provides isolation, using pip installer performs the same installation on the container layers without conflicts.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The following issue and PRs provide a context as to why Pipfile and Pipfile.lock were added:

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The point here is about the conflicts with respect to package installation path. The package management is still being managed by pipenv with usage of pipfile and pipfile.lock. Using PIPENV_SYSTEM=1 enables installing the packages outside the virtual env set by pipenv. This resorts to using underlying system's python version than the desired version with which virtual env is instantiated.

&& yum install -y python3.12 \
&& yum install -y libffi-devel openssl-devel gcc python3.12-devel python3.12-pip python3.12-setuptools \
&& pip3.12 install --upgrade pip~=24.2 \
&& pip3.12 install pipenv \
&& pipenv requirements > requirements.txt \
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using pipenv for now to reuse piplock.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think creating the requirements.txt file while building the image may have some issues while reproducing it locally as was the ask in operator-framework/operator-sdk#4237.
We can have fixed dependencies list beforehand and then use it while building the image. This can be achieved in 2 ways:

  • Either we continue with the same method and generate the Pipfile.lock as mentioned here:

    ansible-operator-plugins/images/ansible-operator/README.md

    Lines 1 to 7 in 42b5d80

    We build the base image using the Dockerfile, which validates the python requirements scaffolding that it copies from this directory.
    To update the requirements (`Pipfile` and `Pipfile.lock`) build and execute the image generated by `pipfile.Dockerfile` like so:
    1. docker build -f ./pipfile.Dockerfile -t pipfile-generator .
    2. docker run --rm -it -v .:/tmp/pip-airlock:Z pipfile-generator
    3. Commit the newly-generated `Pipfile.lock` file (NB: this directory is in root .gitignore file, so you must `git add -f`)
  • Or we generate the requirements.txt file along with Pipfile.lock and then use the requirements.txt file during building the image.

Copy link
Contributor Author

@TrilokGeer TrilokGeer Sep 20, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Or we generate the requirements.txt file along with Pipfile.lock and then use the requirements.txt file during building the image.

The requirements.txt is generated based on Pipfile.lock, refer https://pipenv.pypa.io/en/latest/cli.html#requirements.
Regarding the issue mentioned, the dependencies are managed via pipfile.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ack. But I think the Pipfile.lock file is not updated after your last change to the Pipfile. I used both the files to generate the requirements.txt.backup file. After that I updated the Pipfile.lock using the latest Pipfile and then generated the requirements.txt. Following is the diff between the files:

# diff -y requirements.txt requirements.txt.backup 
-i https://pypi.org/simple					-i https://pypi.org/simple
annotated-types==0.7.0; python_version >= '3.8'			annotated-types==0.7.0; python_version >= '3.8'
							      >	ansible==10.4.0; python_version >= '3.10'
ansible-core==2.17.4; python_version >= '3.10'			ansible-core==2.17.4; python_version >= '3.10'
ansible-runner==2.4.0; python_version >= '3.9'			ansible-runner==2.4.0; python_version >= '3.9'
ansible-runner-http==1.0.0					ansible-runner-http==1.0.0
authlib==1.3.2; python_version >= '3.8'				authlib==1.3.2; python_version >= '3.8'
cachetools==5.5.0; python_version >= '3.7'			cachetools==5.5.0; python_version >= '3.7'
certifi==2024.8.30; python_version >= '3.6'			certifi==2024.8.30; python_version >= '3.6'
cffi==1.17.1; platform_python_implementation != 'PyPy'		cffi==1.17.1; platform_python_implementation != 'PyPy'
charset-normalizer==3.3.2; python_full_version >= '3.7.0'	charset-normalizer==3.3.2; python_full_version >= '3.7.0'
click==8.1.7; python_version >= '3.7'				click==8.1.7; python_version >= '3.7'
cryptography==43.0.1; python_version >= '3.7'			cryptography==43.0.1; python_version >= '3.7'
docutils==0.21.2; python_version >= '3.9'			docutils==0.21.2; python_version >= '3.9'
dparse==0.6.4b0; python_version >= '3.7'			dparse==0.6.4b0; python_version >= '3.7'
filelock==3.12.4; python_version >= '3.8'			filelock==3.12.4; python_version >= '3.8'
google-auth==2.35.0; python_version >= '3.7'		      |	google-auth==2.34.0; python_version >= '3.7'
idna==3.10; python_version >= '3.6'			      |	idna==3.8; python_version >= '3.6'
jinja2==3.1.4; python_version >= '3.7'				jinja2==3.1.4; python_version >= '3.7'
kubernetes==29.0.0; python_version >= '3.6'			kubernetes==29.0.0; python_version >= '3.6'
lockfile==0.12.2						lockfile==0.12.2
markdown-it-py==3.0.0; python_version >= '3.8'			markdown-it-py==3.0.0; python_version >= '3.8'
markupsafe==2.1.5; python_version >= '3.7'			markupsafe==2.1.5; python_version >= '3.7'
marshmallow==3.22.0; python_version >= '3.8'			marshmallow==3.22.0; python_version >= '3.8'
mdurl==0.1.2; python_version >= '3.7'				mdurl==0.1.2; python_version >= '3.7'
oauthlib==3.2.2; python_version >= '3.6'			oauthlib==3.2.2; python_version >= '3.6'
packaging==24.1; python_version >= '3.8'			packaging==24.1; python_version >= '3.8'
pexpect==4.9.0							pexpect==4.9.0
psutil==6.0.0; python_version >= '2.7' and python_version not	psutil==6.0.0; python_version >= '2.7' and python_version not
ptyprocess==0.7.0						ptyprocess==0.7.0
pyasn1==0.6.1; python_version >= '3.8'				pyasn1==0.6.1; python_version >= '3.8'
pyasn1-modules==0.4.1; python_version >= '3.8'			pyasn1-modules==0.4.1; python_version >= '3.8'
pycparser==2.22; python_version >= '3.8'			pycparser==2.22; python_version >= '3.8'
pydantic==2.9.2; python_version >= '3.8'		      |	pydantic==2.9.1; python_version >= '3.8'
pydantic-core==2.23.4; python_version >= '3.8'		      |	pydantic-core==2.23.3; python_version >= '3.8'
pygments==2.18.0; python_version >= '3.8'			pygments==2.18.0; python_version >= '3.8'
python-daemon==3.0.1; python_version >= '3'			python-daemon==3.0.1; python_version >= '3'
python-dateutil==2.9.0.post0; python_version >= '2.7' and pyt	python-dateutil==2.9.0.post0; python_version >= '2.7' and pyt
pyyaml==6.0.2; python_version >= '3.8'				pyyaml==6.0.2; python_version >= '3.8'
requests==2.31.0; python_version >= '3.7'			requests==2.31.0; python_version >= '3.7'
requests-oauthlib==2.0.0; python_version >= '3.4'		requests-oauthlib==2.0.0; python_version >= '3.4'
requests-unixsocket==0.3.0					requests-unixsocket==0.3.0
resolvelib==1.0.1						resolvelib==1.0.1
rich==13.8.1; python_full_version >= '3.7.0'			rich==13.8.1; python_full_version >= '3.7.0'
rsa==4.9; python_version >= '3.6' and python_version < '4'	rsa==4.9; python_version >= '3.6' and python_version < '4'
ruamel.yaml==0.18.6; python_version >= '3.7'			ruamel.yaml==0.18.6; python_version >= '3.7'
ruamel.yaml.clib==0.2.8; python_version < '3.13' and platform	ruamel.yaml.clib==0.2.8; python_version < '3.13' and platform
safety==3.2.7; python_version >= '3.7'				safety==3.2.7; python_version >= '3.7'
safety-schemas==0.0.5; python_version >= '3.7'			safety-schemas==0.0.5; python_version >= '3.7'
setuptools==75.1.0; python_version >= '3.8'		      |	setuptools==74.1.2; python_version >= '3.8'
shellingham==1.5.4; python_version >= '3.7'			shellingham==1.5.4; python_version >= '3.7'
six==1.16.0; python_version >= '2.7' and python_version not i	six==1.16.0; python_version >= '2.7' and python_version not i
typer==0.12.5; python_version >= '3.7'				typer==0.12.5; python_version >= '3.7'
typing-extensions==4.12.2; python_version >= '3.8'		typing-extensions==4.12.2; python_version >= '3.8'
urllib3==1.26.20; python_version >= '2.7' and python_version 	urllib3==1.26.20; python_version >= '2.7' and python_version 
websocket-client==1.8.0; python_version >= '3.8'		websocket-client==1.8.0; python_version >= '3.8'

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, good catch

&& pip3.12 install -r requirements.txt --prefix /usr/local \
# NOTE: This ignored vulnerability (70612) was detected in jinja2, \
# but the vulnerability is disputed and may never be fixed. See: \
# - https://github.com/advisories/GHSA-f6pv-j8mr-w6rr \
Expand All @@ -33,12 +35,12 @@ RUN set -e && yum clean all && rm -rf /var/cache/yum/* \
# NOTE: This ignored vulnerability (71064) was detected in requests, \
# but the upgraded version doesn't support the use case (protocol we are using).\
# Ref: https://github.com/operator-framework/ansible-operator-plugins/pull/67#issuecomment-2189164688
&& pipenv check --ignore 70612 --ignore 71064 \
&& yum remove -y gcc libffi-devel openssl-devel python39-devel \
&& safety check --ignore 70612 --ignore 71064 \
&& yum remove -y gcc libffi-devel openssl-devel python3.12-devel \
&& yum clean all \
&& rm -rf /var/cache/yum

FROM registry.access.redhat.com/ubi8/ubi:8.9-1107 as base
FROM registry.access.redhat.com/ubi9/ubi:9.4-1214 as base
ARG TARGETARCH

# Label this image with the repo and commit that built it, for freshmaking purposes.
Expand All @@ -53,14 +55,15 @@ RUN mkdir -p /etc/ansible \

RUN set -e && yum clean all && rm -rf /var/cache/yum/* \
&& yum update -y \
&& yum install -y python39-pip python39-setuptools \
&& pip3 install --upgrade pip~=23.3.2 \
&& pip3 install pipenv==2023.11.15 \
&& yum install -y python3.12 \
&& yum install -y python3.12-pip python3.12-setuptools \
&& pip3.12 install --upgrade pip~=24.2 \
&& pip3.12 install pipenv==2024.0.1 \
&& yum clean all \
&& rm -rf /var/cache/yum

COPY --from=basebuilder /usr/local/lib64/python3.9/site-packages /usr/local/lib64/python3.9/site-packages
COPY --from=basebuilder /usr/local/lib/python3.9/site-packages /usr/local/lib/python3.9/site-packages
COPY --from=basebuilder /usr/local/lib64/python3.12/site-packages /usr/local/lib64/python3.12/site-packages
COPY --from=basebuilder /usr/local/lib/python3.12 /usr/local/lib/python3.12
COPY --from=basebuilder /usr/local/bin /usr/local/bin

ENV TINI_VERSION=v0.19.0
Expand Down
12 changes: 7 additions & 5 deletions images/ansible-operator/Pipfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,16 @@ verify_ssl = true
name = "pypi"

[packages]
ansible-runner = "~=2.3.6"
ansible-core = "==2.17.4"
ansible-runner = "~=2.4.0"
ansible-runner-http = "~=1.0.0"
ansible-core = "~=2.15.12"
urllib3 = "~=1.26"
urllib3 = "~=1.26.2"
kubernetes = "==29.0.0"
requests = "~=2.31.0"
safety = "==3.2.7"
PyYAML = "==6.0.2"
Comment on lines +12 to +13
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any specific reason why safety and PyYAML are pinned to the specific versions? I think these will be automatically upgraded to the latest version when we re-generate the Pipfile.lock from the Pipfile when they are not pinned to a specific version.

Copy link
Contributor Author

@TrilokGeer TrilokGeer Sep 24, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These packages are pinned to have dependent pinned version of these packages as pip freeze is not being used. Pip freeze will result in a list of packages that generate version conflicts. Hence, using this way of installation of these packages pinned with latest possible versions.

requests = "==2.31.0"

[dev-packages]

[requires]
python_version = "3.9"
python_version = "3.12"
Loading