Using Nexus as a proxy registry for a disconnected OpenShift install

[ElCoyote27]

Sonatype Nexus Repository OSS users can file an issue here.

I’m attempting to use Nexus as a local proxy registry for deploying OCP clusters.

I’ve configured upstream docker proxy repos to quay.io, registry.redhat.io and a few others.
Then, I’ve aggregated those repos (each with their pull secrets) under a group hosted on Nexus on port 5000.

This works fine for image-based downloads.
Upstream:

podman pull quay.io/openshift-release-dev/ocp-release:4.16.5-x86_64
My registry:

podman pull registry.lasthome.solace.krynn:5000/openshift-release-dev/ocp-release:4.16.5-x86_64
Both of these work without issues.

However, if I try to deploy Openshift using Nexus as my registry, I get errors for the sha256 images (digests?), e.g:
This one works (using upstream):

podman pull quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e4b8d40d25ba75162a201b45ff09dea4dd9fb13d078bbd6dd09276266df6842
but this one fails:

podman pull registry.lasthome.solace.krynn:5000/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e4b8d40d25ba75162a201b45ff09dea4dd9fb13d078bbd6dd09276266df6842
and it reports: “manifest unknown”

In the nexus outbound Log, I can then see 401 errors such as these:

[26/Aug/2024:12:36:09 +0000] 7742700|uhc-1L9AybTZWMqYYhqpfGHL945CLJ0 "GET https://quay.io/v2/openshift-release-dev/ocp-v4.0-art-dev/manifests/sha256:1e4b8d40d25ba75162a201b45ff09dea4dd9fb13d078bbd6dd09276266df6842 HTTP/1.1" 401 112 299 "Nexus/3.71.0-06 (OSS; Linux; 4.18.0-553.16.1.el8_10.x86_64; amd64; 17.0.12)" [qtp565611798-4260]
[26/Aug/2024:12:36:11 +0000] 7742700|uhc-1L9AybTZWMqYYhqpfGHL945CLJ0 "GET https://quay.io/v2/openshift-release-dev/ocp-v4.0-art-dev/manifests/sha256:1e4b8d40d25ba75162a201b45ff09dea4dd9fb13d078bbd6dd09276266df6842 HTTP/1.1" 401 112 303 "Nexus/3.71.0-06 (OSS; Linux; 4.18.0-553.16.1.el8_10.x86_64; amd64; 17.0.12)" [qtp565611798-4260]
Is this doable? Am I doing something wrong?

For reference, this is what I am using in my install-config:

imageContentSources:
- mirrors:
  - registry.lasthome.solace.krynn:5000/openshift-release-dev/ocp-v4.0-art-dev
  source: quay.io/openshift-release-dev/ocp-v4.0-art-dev
- mirrors:
  - registry.lasthome.solace.krynn:5000/openshift-release-dev/ocp-release
  source: quay.io/openshift-release-dev/ocp-release

I’ve reached out to OpenShift support and they said that the only way this could fail with a 401 on quay.io is if the requests aren’t properly authenticated:

The v2/auth endpoint on Quay will only return a 401 if you have bad credentials. There would be no other reason to do so.
Without credentials:

# podman pull quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e4b8d40d25ba75162a201b45ff09dea4dd9fb13d078bbd6dd09276266df6842
Trying to pull quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e4b8d40d25ba75162a201b45ff09dea4dd9fb13d078bbd6dd09276266df6842...
Error: initializing source docker://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e4b8d40d25ba75162a201b45ff09dea4dd9fb13d078bbd6dd09276266df6842: reading manifest sha256:1e4b8d40d25ba75162a201b45ff09dea4dd9fb13d078bbd6dd09276266df6842 in quay.io/openshift-release-dev/ocp-v4.0-art-dev: unauthorized: access to the requested resource is not authorized

With credentials:

# podman pull --authfile pull-secret.json quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e4b8d40d25ba75162a201b45ff09dea4dd9fb13d078bbd6dd09276266df6842
Trying to pull quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e4b8d40d25ba75162a201b45ff09dea4dd9fb13d078bbd6dd09276266df6842...
Getting image source signatures
Copying blob 8deeda9a2ea3 done   | 
Copying blob 68957586c3c2 done   | 
Copying blob ca1636478fe5 done   | 
Copying blob 312897d9e4ab done   | 
Copying config e1f53e55a0 done   | 
Writing manifest to image destination
e1f53e55a022852fd7f988ce0a92f262adf73de64a1ed99700040f8df0f098ae

[Thulium-Drake]

I seem to running into the same issue trying to configure Nexus as a proxy for generating Device Edge images with Microshift. Could you please share your configuration in Nexus? I currently have this:

However, whether I put in my credentials or not, it's not working (also not when using a file with pull-secrets).

I'm missing something, but I don't know what 😅

(EDIT: I only have issues with quay.io, all redhat.com/io registries are working fine)

[ElCoyote27]

@Thulium-Drake Hi, I think you're very close but for some reason your screenshot doesn't look like the proxy configuration I have in my Nexus (3.73.0).
First of all, for OpenShift, get the pull secret and decrypt it.
I use this simple script to pretty print the content:

$ cat extract.sh
#!/bin/bash
#
[ "$BASH" ] && function whence
{
        type -p "$@"
}
#
PATH_SCRIPT="$(cd $(/usr/bin/dirname $(whence -- $0 || echo $0));pwd)"

cat ${PATH_SCRIPT}/pull-secret.txt|jq .|grep -B1 auth.:|xargs -n5|awk '{ print $1,$4}'|while read site auth
do
        my_login="$(echo $auth|base64 -id|cut -f1 -d:)"
        my_pwd="$(echo $auth|base64 -id|cut -f2 -d: |sed -e 's@,$@@')"
        my_url="$(echo $site)"
        echo -e "\n * https://${my_url}"
        echo -e "\tLOGIN: $my_login"
        echo -e "\tPWD: $my_pwd"
done

This should give you two logins and 4 registry URLs (obfuscated):

 * https://cloud.openshift.com:
        LOGIN: openshift-release-dev+[....]clj0
        PWD: EH[....]ADDFN5S3

 * https://quay.io:
        LOGIN: openshift-release-dev+[....]clj0
        PWD: EH[....]ADDFN5S3

 * https://registry.connect.redhat.com:
        LOGIN: 7742700|uhc[....]5CLJ0
        PWD: eyJ[....]Kv8M

 * https://registry.redhat.io:
        LOGIN: 7742700|uhc[....]5CLJ0
        PWD: eyJh[....]xNDR

The way I am doing this is by giving each repo its own private port and aggregate all of them under a group called 'redhat'.

In the ADMIN UI side of Nexus, my config looks like this:

Now, for each of the repos, I define a specific port and enter the credentials obtained from the pull secret, like this:

and at the end the authentication from the pull secret:

Repeat for each upstream registry and aggregate all of them under a group registry (that one is on 18000 which gets TLS'ed to 5000):

It used to work fine but in the recent months it ceased working for some images (hence this nexus issue).

[ElCoyote27]

When I 'browse' the 'redhat' repository, I can see content cached from various upstream repos: