CFE-1124: Install ansible using pip instead of rpm #20

arkadeepsen · 2024-08-28T16:46:33Z

Description of the change:
This PR updates the Dockerfile to use pip instead of rpm to install ansible dependencies.

Following changes are made in this PR:

Add Dockerfile.requirements for generating the requirements.txt and requirments-build.txt files from the upstream Pipfile
Add make target to generate the requirements. Add make target to check if the current requirements.txt and requirements-build.txt files are up-to-date
Update Dockerfile to use cachito speicific env vars for installing the python modules using pip
Remove unused shell scripts and inline the user creation script in the Dockerfile
Update base images to rhel9 in Dockerfile

Motivation for the change:

In rhel8, ansible-core rpm uses python3.12, while the other ansible dependencies (ansible-runner, ansible-runner-http, python3-kubenetes, python3-openshit) use python3.6. This is causing ansible-runner to not find ansible during runtime of the operator created using the ansible-operator base image.

In rhel9, the various dependencies of ansible (ansible-runner, ansible-runner-http, python3-kubenetes, python3-openshit) will not be available as part of rpm packages.

For the above reasons ansible is being installed using pip instead of rpm.

openshift-ci · 2024-08-28T16:51:47Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: arkadeepsen

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [arkadeepsen]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

arkadeepsen · 2024-09-05T12:47:32Z

/test all

arkadeepsen · 2024-09-05T16:33:47Z

/retest

arkadeepsen · 2024-09-05T17:45:15Z

/retest

openshift-ci-robot · 2024-09-09T04:54:58Z

@arkadeepsen: This pull request references CFE-1124 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the sub-task to target the "4.18.0" version, but no target version was set.

In response to this:

Description of the change:

Motivation for the change:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot · 2024-09-09T14:56:06Z

@arkadeepsen: This pull request references CFE-1124 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the sub-task to target the "4.18.0" version, but no target version was set.

In response to this:

Description of the change:
This PR updates the Dockerfile to use pip instead of rpm to install ansible dependencies.

Following changes are made in this PR:

Add Dockerfile.requirements for generating the requirements.txt and requirments-build.txt files from the upstream Pipfile

Add make target to generate the requirements. Add make target to check if the current requirements.txt and requirements-build.txt files are up-to-date

Update Dockerfile to use cachito speicific env vars for installing the python modules using pip

Remove unused shell scripts and inline the user creation script in the Dockerfile

Update base images to rhel9 in Dockerfile

Motivation for the change:

In rhel8, ansible-core rpm uses python3.12, while the other ansible dependencies (ansible-runner, ansible-runner-http, python3-kubenetes, python3-openshit) use python3.6. This is causing ansible-runner to not find ansible during runtime of the operator created using the ansible-operator base image.

In rhel9, the various dependencies of ansible (ansible-runner, ansible-runner-http, python3-kubenetes, python3-openshit) will not be available as part of rpm packages.

For the above reasons ansible is being installed using pip instead of rpm.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

TrilokGeer · 2024-09-19T12:24:22Z

openshift/Dockerfile.requirements

+RUN python3 -m pip install pipenv==2023.11.15 \
+    && python3 -m pip install pip-tools \
+    && pipenv install --deploy \
+    && pipenv run pip freeze --all  > ./requirements.in \


It is better to have check for cve using pipenv check or safety to fail-fast before the building the requirements.

How do we proceed if there are CVEs reported by pipenv check? The Pipfile and the Pipfile.lock are what we have from upstream. Do we go upstream first and update the files and then pull it downstream?

Depends on the timeline commitments. If the downstream release can wait till there is a release from upstream, then upstream fixes can rebased. Otherwise, the upstream carry commits can be created. Once the fix is available from upstream, the carry PRs can be dropped.

Found the following vulnerabilities after using pipenv check --ignore 70612 --ignore 71064:

20.87 Checking Pipfile.lock packages for vulnerabilities... 20.87 Notice: Ignoring Vulnerabilities 70612, 71064 22.84 +==============================================================================+ 22.84 22.84 /$$$$$$ /$$ 22.84 /$$__ $$ | $$ 22.84 /$$$$$$$ /$$$$$$ | $$ \__//$$$$$$ /$$$$$$ /$$ /$$ 22.84 /$$_____/ |____ $$| $$$$ /$$__ $$|_ $$_/ | $$ | $$ 22.84 | $$$$$$ /$$$$$$$| $$_/ | $$$$$$$$ | $$ | $$ | $$ 22.84 \____ $$ /$$__ $$| $$ | $$_____/ | $$ /$$| $$ | $$ 22.84 /$$$$$$$/| $$$$$$$| $$ | $$$$$$$ | $$$$/| $$$$$$$ 22.84 |_______/ \_______/|__/ \_______/ \___/ \____ $$ 22.84 /$$ | $$ 22.84 | $$$$$$/ 22.84 by pyup.io \______/ 22.84 22.84 +==============================================================================+ 22.84 22.84 REPORT 22.84 22.84 Safety v2.3.2 is scanning for Vulnerabilities... 22.84 Scanning dependencies in your files: 22.84 22.84 -> /tmp/-x-v5uFv0wqfu2bdb_requirements.txt 22.84 22.84 Found and scanned 37 packages 22.84 Timestamp 2024-09-19 16:09:39 22.84 5 vulnerabilities found 22.84 2 vulnerabilities ignored 22.84 22.84 +==============================================================================+ 22.84 VULNERABILITIES FOUND 22.84 +==============================================================================+ 22.84 22.84 -> Vulnerability found in certifi version 2024.2.2 22.84 Vulnerability ID: 72083 22.84 Affected spec: >=2021.05.30,<2024.07.04 22.84 ADVISORY: Certifi affected versions recognized root certificates from 22.84 GLOBALTRUST. Certifi patch removes these root certificates from the root... 22.84 CVE-2024-39689 22.84 For more information, please visit 22.84 https://data.safetycli.com/v/72083/742 22.84 22.84 22.84 -> Vulnerability found in cryptography version 42.0.7 22.84 Vulnerability ID: 71681 22.84 Affected spec: <42.0.8 22.84 ADVISORY: The `cryptography` library has updated its BoringSSL and 22.84 OpenSSL dependencies in CI due to a security concern. Specifically, the... 22.84 CVE-2024-4603 22.84 For more information, please visit 22.84 https://data.safetycli.com/v/71681/742 22.84 22.84 22.84 -> Vulnerability found in jinja2 version 3.1.4 22.84 Vulnerability ID: 70612 22.84 This vulnerability is being ignored. 22.84 For more information, please visit 22.84 https://data.safetycli.com/v/70612/742 22.84 22.84 22.84 -> Vulnerability found in requests version 2.31.0 22.84 Vulnerability ID: 71064 22.84 This vulnerability is being ignored. 22.84 For more information, please visit 22.84 https://data.safetycli.com/v/71064/742 22.84 22.84 22.84 -> Vulnerability found in setuptools version 69.5.1 22.84 Vulnerability ID: 72236 22.84 Affected spec: <70.0.0 22.84 ADVISORY: Affected versions of Setuptools allow for remote code 22.84 execution via its download functions. These functions, which are used to... 22.84 CVE-2024-6345 22.84 For more information, please visit 22.84 https://data.safetycli.com/v/72236/742 22.84 22.84 22.84 -> Vulnerability found in urllib3 version 1.26.18 22.84 Vulnerability ID: 71608 22.84 Affected spec: <=1.26.18 22.84 ADVISORY: Urllib3's ProxyManager ensures that the Proxy-Authorization 22.84 header is correctly directed only to configured proxies. However, when... 22.84 CVE-2024-37891 22.84 For more information, please visit 22.84 https://data.safetycli.com/v/71608/742 22.84 22.84 22.84 -> Vulnerability found in zipp version 3.18.1 22.84 Vulnerability ID: 72132 22.84 Affected spec: <3.19.1 22.84 ADVISORY: A Denial of Service (DoS) vulnerability exists in the 22.84 jaraco/zipp library. The vulnerability is triggered when processing a... 22.84 CVE-2024-5569 22.84 For more information, please visit 22.84 https://data.safetycli.com/v/72132/742 22.84 22.84 Scan was completed. 5 vulnerabilities were found. 2 vulnerabilities from 2 22.84 packages were ignored. 22.84 22.84 +==============================================================================+ 22.84 REMEDIATIONS 22.84 22.84 5 vulnerabilities were found in 5 packages. For detailed remediation & fix 22.84 recommendations, upgrade to a commercial license. 22.84 22.84 +==============================================================================+

How should we proceed now?

I addition to this we have to install cryptography using rpm as it cannot be built using cachito. Following are the versions available for cryptography:

bash-5.1$ yum --showduplicates list python3-cryptography Last metadata expiration check: 0:03:40 ago on Thu Sep 19 17:10:30 2024. Installed Packages python3-cryptography.x86_64 36.0.1-2.el9 @ubi-9-appstream-rpms Available Packages python3-cryptography.x86_64 36.0.1-2.el9 ubi-9-appstream-rpms bash-5.1$ yum --showduplicates list python3.11-cryptography Last metadata expiration check: 0:03:58 ago on Thu Sep 19 17:10:30 2024. Available Packages python3.11-cryptography.x86_64 37.0.2-6.el9 ubi-9-appstream-rpms bash-5.1$ yum --showduplicates list python3.12-cryptography Last metadata expiration check: 0:04:01 ago on Thu Sep 19 17:10:30 2024. Available Packages python3.12-cryptography.x86_64 41.0.7-1.el9 ubi-9-appstream-rpms

Added the commit c42e6b6 to fix the vulnerabilities issues that are reported in the above comment. Using the updated Pipfile and Pipfile.lock to generate the different requirements files.

We can drop c42e6b6 whenever we pull in the changes from upstream after operator-framework/ansible-operator-plugins#92 is merged.

Updated the Pipfile to pin urllib3 version to 2.31.0 because of this issue:

ansible-operator-plugins relying on abandoned python packages operator-framework/ansible-operator-plugins#86

The requirements.txt file is also updated

Updated the Pipfile to pin urllib3 version to 2.31.0 because of this issue:

Did you mean requests = "==2.31.0"? Because urllib3 is at urllib3 = "~=1.26.19"
https://github.com/openshift/ansible-operator-plugins/pull/20/files#diff-85c2285c39846cfb1ab26c4373213d7c91c4705e10d590b41c4d9503127fe812L10-L12

Ah yes. My bad. It's requests package which is pinned to 2.31.0.

TrilokGeer · 2024-09-24T11:11:59Z

/lgtm

chiragkyal · 2024-09-24T11:17:06Z

Adding some documentation with the steps to update the requirements or changing the dependent packages would help.

…ages with vulnerabilities

…mage Add Dockerfile.requirements for generating the requirements.txt and requirments-build.txt files from the upstream Pipfile Add make target to generate the requirements Add make target to check if the current requirements.txt and requirements-build.txt files are up-to-date Update Dockerfile to use cachito specific env vars for installing the python modules using pip Remove unused shell scripts and inline the user creation script in the Dockerfile Update base images to rhel9 in Dockerfile

arkadeepsen · 2024-09-24T12:00:58Z

/hold Until all the rpm packages are finalized.

/hold cancel
Removing the hold as the rpm packages are finalized.

TrilokGeer · 2024-09-24T12:02:45Z

/lgtm

openshift-ci · 2024-09-24T12:52:08Z

@arkadeepsen: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

KeenonLee

/label qe-approved

openshift-ci-robot · 2024-09-25T05:21:18Z

@arkadeepsen: This pull request references CFE-1124 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the sub-task to target the "4.18.0" version, but no target version was set.

In response to this:

Description of the change:
This PR updates the Dockerfile to use pip instead of rpm to install ansible dependencies.

Following changes are made in this PR:

Add Dockerfile.requirements for generating the requirements.txt and requirments-build.txt files from the upstream Pipfile

Add make target to generate the requirements. Add make target to check if the current requirements.txt and requirements-build.txt files are up-to-date

Update Dockerfile to use cachito speicific env vars for installing the python modules using pip

Remove unused shell scripts and inline the user creation script in the Dockerfile

Update base images to rhel9 in Dockerfile

Motivation for the change:

In rhel8, ansible-core rpm uses python3.12, while the other ansible dependencies (ansible-runner, ansible-runner-http, python3-kubenetes, python3-openshit) use python3.6. This is causing ansible-runner to not find ansible during runtime of the operator created using the ansible-operator base image.

In rhel9, the various dependencies of ansible (ansible-runner, ansible-runner-http, python3-kubenetes, python3-openshit) will not be available as part of rpm packages.

For the above reasons ansible is being installed using pip instead of rpm.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

arkadeepsen · 2024-09-25T05:35:01Z

/cherrypick release-4.17

openshift-cherrypick-robot · 2024-09-25T05:35:40Z

@arkadeepsen: new pull request created: #21

In response to this:

/cherrypick release-4.17

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

openshift-bot · 2024-09-25T09:40:27Z

[ART PR BUILD NOTIFIER]

Distgit: openshift-enterprise-ansible-operator
This PR has been included in build openshift-enterprise-ansible-operator-container-v4.18.0-202409250812.p0.gd31f0a9.assembly.stream.el9.
All builds following this will include this PR.

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 28, 2024

openshift-ci bot requested review from grokspawn and oceanc80 August 28, 2024 16:51

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 28, 2024

arkadeepsen force-pushed the ansible-core branch 3 times, most recently from 2ba27f3 to f429941 Compare August 29, 2024 11:14

arkadeepsen force-pushed the ansible-core branch 2 times, most recently from 56b540d to 055157e Compare September 5, 2024 12:08

arkadeepsen force-pushed the ansible-core branch 2 times, most recently from 39f84f0 to a9dfd45 Compare September 9, 2024 04:52

arkadeepsen changed the title ~~[WIP] DNM: Install ansible-core instead of ansible~~ CFE-1124: Install ansible-core instead of ansible Sep 9, 2024

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 9, 2024

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Sep 9, 2024

arkadeepsen force-pushed the ansible-core branch 6 times, most recently from 66c3432 to cb12ea5 Compare September 9, 2024 13:08

arkadeepsen changed the title ~~CFE-1124: Install ansible-core instead of ansible~~ CFE-1124: Install ansible using pip instead of rpm Sep 9, 2024

arkadeepsen force-pushed the ansible-core branch 3 times, most recently from 01e052b to 4f8cfa9 Compare September 10, 2024 08:33

arkadeepsen force-pushed the ansible-core branch from ce8a18a to d0fe4e7 Compare September 19, 2024 09:07

TrilokGeer reviewed Sep 19, 2024

View reviewed changes

arkadeepsen force-pushed the ansible-core branch 7 times, most recently from 560c156 to 94d4d98 Compare September 20, 2024 15:01

openshift-ci bot assigned TrilokGeer Sep 24, 2024

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Sep 24, 2024

arkadeepsen added 2 commits September 24, 2024 16:59

UPSTREAM: <carry>: Update Pipfile and Pipfile.lock for upgrading pack…

a0a6116

…ages with vulnerabilities

arkadeepsen force-pushed the ansible-core branch from 532bfba to d930942 Compare September 24, 2024 11:36

openshift-ci bot removed the lgtm Indicates that a PR is ready to be merged. label Sep 24, 2024

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Sep 24, 2024

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Sep 24, 2024

KeenonLee reviewed Sep 25, 2024

View reviewed changes

openshift-ci bot added the qe-approved Signifies that QE has signed off on this PR label Sep 25, 2024

openshift-merge-bot bot merged commit d31f0a9 into openshift:main Sep 25, 2024
5 checks passed

openshift-cherrypick-robot mentioned this pull request Sep 25, 2024

[release-4.17] OCPBUGS-42410: Install ansible using pip instead of rpm #21

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CFE-1124: Install ansible using pip instead of rpm #20

CFE-1124: Install ansible using pip instead of rpm #20

arkadeepsen commented Aug 28, 2024 •

edited

Loading

openshift-ci bot commented Aug 28, 2024

arkadeepsen commented Sep 5, 2024

arkadeepsen commented Sep 5, 2024

arkadeepsen commented Sep 5, 2024

openshift-ci-robot commented Sep 9, 2024 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented Sep 9, 2024 •

edited by openshift-ci bot

Loading

TrilokGeer Sep 19, 2024

arkadeepsen Sep 19, 2024

TrilokGeer Sep 19, 2024

arkadeepsen Sep 19, 2024

arkadeepsen Sep 19, 2024 •

edited

Loading

arkadeepsen Sep 20, 2024

arkadeepsen Sep 20, 2024

arkadeepsen Sep 20, 2024 •

edited

Loading

chiragkyal Sep 23, 2024

arkadeepsen Sep 23, 2024

TrilokGeer commented Sep 24, 2024

chiragkyal commented Sep 24, 2024 •

edited

Loading

arkadeepsen commented Sep 24, 2024

TrilokGeer commented Sep 24, 2024

openshift-ci bot commented Sep 24, 2024

KeenonLee left a comment

openshift-ci-robot commented Sep 25, 2024 •

edited by openshift-ci bot

Loading

arkadeepsen commented Sep 25, 2024

openshift-cherrypick-robot commented Sep 25, 2024

openshift-bot commented Sep 25, 2024

CFE-1124: Install ansible using pip instead of rpm #20

CFE-1124: Install ansible using pip instead of rpm #20

Conversation

arkadeepsen commented Aug 28, 2024 • edited Loading

openshift-ci bot commented Aug 28, 2024

arkadeepsen commented Sep 5, 2024

arkadeepsen commented Sep 5, 2024

arkadeepsen commented Sep 5, 2024

openshift-ci-robot commented Sep 9, 2024 • edited by openshift-ci bot Loading

openshift-ci-robot commented Sep 9, 2024 • edited by openshift-ci bot Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arkadeepsen Sep 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arkadeepsen Sep 20, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TrilokGeer commented Sep 24, 2024

chiragkyal commented Sep 24, 2024 • edited Loading

arkadeepsen commented Sep 24, 2024

TrilokGeer commented Sep 24, 2024

openshift-ci bot commented Sep 24, 2024

KeenonLee left a comment

Choose a reason for hiding this comment

openshift-ci-robot commented Sep 25, 2024 • edited by openshift-ci bot Loading

arkadeepsen commented Sep 25, 2024

openshift-cherrypick-robot commented Sep 25, 2024

openshift-bot commented Sep 25, 2024

arkadeepsen commented Aug 28, 2024 •

edited

Loading

openshift-ci-robot commented Sep 9, 2024 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented Sep 9, 2024 •

edited by openshift-ci bot

Loading

arkadeepsen Sep 19, 2024 •

edited

Loading

arkadeepsen Sep 20, 2024 •

edited

Loading

chiragkyal commented Sep 24, 2024 •

edited

Loading

openshift-ci-robot commented Sep 25, 2024 •

edited by openshift-ci bot

Loading