Add example of publishing PyPI package to Local Python repository #22

jvroberts · 2023-07-20T20:46:10Z

An example of how to make PyPI packages available from an offline Package Manager instance, until Package Manager supports air-gapped/offline PyPI repositories.

Adds an example bash script that will:

Download a specified package source tarball from PyPI and its associated binary wheels.
Upload those package files to a local Package Manager instance using twine

jmwoliver

This looks great! I tested it out and made a note of a few things I noticed.

What is the likelihood of an air-gapped customer being able to use this? Would they have pypi.org blocked on their network, making them unable to pull down packages like this?

pypi-to-local-python/add-pypi-package.sh

shepherdjerred · 2023-07-20T21:35:01Z

Looks reasonable to me once the issues Jacob brought up are addressed.

jvroberts · 2023-07-21T00:52:32Z

What is the likelihood of an air-gapped customer being able to use this? Would they have pypi.org blocked on their network, making them unable to pull down packages like this?

That's a good question, but technically you could run this from some machine in a DMZ that does have access to PyPI, but also has access to the internal PPM and can upload the packages to there, I guess. You could also adapt the example to separately pull the packages you want then copy them manually and push them. It's a starting point at least!

jonyoder

This looks really neat! The only suggestion I'd have is to write another alternative script that would let you do the same thing for an entire repo; then a customer could follow this workflow:

Install a temporary PPM instance with a trial license (that has access to the internet).
Create a Curated-PyPI source/repo and populate it with the packages they need.
Run the script against their curated repo to download all the repo packages and upload them to their real PPM server.

jonyoder

Here's a round of some feedback.

jonyoder · 2023-07-21T13:33:44Z

pypi-to-local-python/add-pypi-package.sh

+echo Downloading package files for "$PACKAGE" from PyPI...
+echo
+
+curl https://pypi.org/pypi/$PACKAGE/json | jq ".releases[$VERSION][] | .url" | xargs -n1 curl --retry 2 -O --output-dir $PKGDIR


Is it a bit sad that we're not defaulting to P3M here? I realize it has a different API, but we could consider supporting both.

Yeah, I originally planned to do that but then the PyPI API was easier. I'd welcome an adaptation to use P3M instead.

jonyoder · 2023-07-21T13:37:55Z

pypi-to-local-python/add-pypi-package.sh

+if [ "$2" = "" ]
+then
+    VERSION=.info.version
+else
+    VERSION=\"$2\"
+fi


You should be able to shorten this entire block to:

VERSION=${2:-.info.version}

I need a bash wizard on retainer... :)

My fees are reasonable. 🤣 (although I wouldn't say I'm a wizard)

Except if $2 is provided, I need it to be double-quoted, but if using the default .info.version I need to be unquoted. What's the cleanest way to make that happen?

I''m sure a real bash wizard would know how, but the smallest I'm coming up with is:

VERSION=$([[ -z $2 ]] && echo ".info.version" || echo "\"$2\"")

Your original version is more readable!

Actually, here's one idea that's not so bad:

VERSION=${2:+\"$2\"} VERSION=${VERSION:-.info.version}

That works nicely!

jonyoder · 2023-07-21T13:40:45Z

pypi-to-local-python/add-pypi-package.sh

+KEEP_PACKAGES=false
+
+# name of the Package Manager python local source to add packages to
+PACKAGEMANAGER_SOURCE=python


Here, and for the two other options above, you should consider making these optional defaults, like this:

# Caller can set TEMPDIR, or default to $TMPDIR or /tmp TEMPDIR=${TEMPDIR:-${TMPDIR:-/tmp}} KEEP_PACKAGES=${KEEP_PACKAGES:-false} PACKAGEMANAGER_SOURCE=${PACKAGEMANAGER_SOURCE:-python}

The $TMPDIR default will automatically honor any environments that use a custom $TMPDIR setting.

With that change, users don't need to edit the script, but if they want something other than the default values, they can simply set env vars for the script, like:

KEEP_PACKAGES=true ./add-pypi-packages.sh appdirs 3.0.4

jonyoder · 2023-07-21T13:42:43Z

pypi-to-local-python/add-pypi-package.sh

+PACKAGEMANAGER_SOURCE=python
+
+# Package Manager address and API token with permission to upload to source
+PACKAGEMANAGER_ADDRESS=http://localhost:4242


Same as above; here we could have this be a default address but allow an env var to override it.

jonyoder · 2023-07-21T13:44:16Z

pypi-to-local-python/add-pypi-package.sh

+
+# Package Manager address and API token with permission to upload to source
+PACKAGEMANAGER_ADDRESS=http://localhost:4242
+PACKAGEMANAGER_TOKEN=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJwYWNrYWdlbWFuYWdlciIsImp0aSI6ImQwOTIxZmJhLTcwNTUtNDU4Ni1iNTkwLWNkZDJiODJjMWI0NiIsImlhdCI6MTY4OTg3MjgzNCwiaXNzIjoicGFja2FnZW1hbmFnZXIiLCJzY29wZXMiOnsic291cmNlcyI6IjUzYmZlNGQyLTExYTUtNGI5Yi1iM2Q3LTc2NjU5YjExYWVlMiJ9fQ.KKTmNw32JM6IM30XCeJbadJSxGw3z6bNW0BqMwSqdus


We should probably discourage setting a token in this script since it's a security risk. Instead, I'd recommend simply checking that this env var is set:

if [[ "$PACKAGEMANAGER_TOKEN" == "" ]]; then echo "You should define the PACKAGEMANAGER_TOKEN environment variable before using this script." fi

If someone really wants the token in this script, they can still modify it and add it, of course.

jonyoder · 2023-07-21T14:09:39Z

pypi-to-local-python/add-pypi-package.sh

+if [ "$KEEP_PACKAGES" = "false" ]
+then
+    rm -rf $PKGDIR
+fi


Cleanup works best as a trap. I'd recommend adding this up on line 50, right below where PKGDIR is set:

PKGDIR=$TEMPDIR/$PACKAGE mkdir -p $PKGDIR cleanup () { if [ "$KEEP_PACKAGES" = "false" ]; then rm -rf $PKGDIR fi } trap cleanup EXIT

This will clean things up even if the script fails due to a network error, 404, etc.

jonyoder · 2023-07-21T14:11:37Z

pypi-to-local-python/add-pypi-package.sh

+echo Downloading package files for "$PACKAGE" from PyPI...
+echo
+
+curl https://pypi.org/pypi/$PACKAGE/json | jq ".releases[$VERSION][] | .url" | xargs -n1 curl --retry 2 -O --output-dir $PKGDIR


What happens if you specify a bad package name? You'll want to verify that the curl call was successful before you continue. Same with the twine upload command below.

I'd recommend using the -f flag to make curl fail if the command isn't successful. Then you can check the exit code, like:

# Get the JSON data from PyPI url=https://pypi.org/pypi/$PACKAGE/json json=$(curl -sf $url) if [[ "$?" -ne 0 ]]; then echo "Error downloading package JSON metadata for $PACKAGE from $url" fi # Download the files echo $json | jq ".releases[$VERSION][] | .url" | xargs -n1 curl --retry 2 -O --output-dir $PKGDIR

Example of PyPI to Local Python

05c6f82

jmwoliver reviewed Jul 20, 2023

View reviewed changes

pypi-to-local-python/add-pypi-package.sh Outdated Show resolved Hide resolved

pypi-to-local-python/add-pypi-package.sh Outdated Show resolved Hide resolved

pypi-to-local-python/add-pypi-package.sh Outdated Show resolved Hide resolved

jvroberts added 3 commits July 20, 2023 15:21

Remove debugging arg

b861fb1

Call twine directly.

db426f9

Add parents to mkdir.

4b9e8f0

jonyoder approved these changes Jul 21, 2023

View reviewed changes

jonyoder reviewed Jul 21, 2023

View reviewed changes

Incorporate review feedback

e8a7ce9

jvroberts merged commit bf72bcf into main Jul 21, 2023
72 checks passed

jvroberts deleted the pypi-to-local branch July 21, 2023 17:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add example of publishing PyPI package to Local Python repository #22

Add example of publishing PyPI package to Local Python repository #22

jvroberts commented Jul 20, 2023

jmwoliver left a comment

shepherdjerred commented Jul 20, 2023 •

edited

Loading

jvroberts commented Jul 21, 2023

jonyoder left a comment

jonyoder left a comment

jonyoder Jul 21, 2023

jvroberts Jul 21, 2023

jonyoder Jul 21, 2023

jvroberts Jul 21, 2023

jonyoder Jul 21, 2023 •

edited

Loading

jvroberts Jul 21, 2023

jonyoder Jul 21, 2023

jonyoder Jul 21, 2023

jvroberts Jul 21, 2023

jonyoder Jul 21, 2023

jonyoder Jul 21, 2023

jonyoder Jul 21, 2023

jonyoder Jul 21, 2023 •

edited

Loading

jonyoder Jul 21, 2023

jonyoder Jul 21, 2023 •

edited

Loading

Add example of publishing PyPI package to Local Python repository #22

Add example of publishing PyPI package to Local Python repository #22

Conversation

jvroberts commented Jul 20, 2023

jmwoliver left a comment

Choose a reason for hiding this comment

shepherdjerred commented Jul 20, 2023 • edited Loading

jvroberts commented Jul 21, 2023

jonyoder left a comment

Choose a reason for hiding this comment

jonyoder left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jonyoder Jul 21, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jonyoder Jul 21, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jonyoder Jul 21, 2023 • edited Loading

Choose a reason for hiding this comment

shepherdjerred commented Jul 20, 2023 •

edited

Loading

jonyoder Jul 21, 2023 •

edited

Loading

jonyoder Jul 21, 2023 •

edited

Loading

jonyoder Jul 21, 2023 •

edited

Loading