OKD Support

[itmwiw]

Hello,
Is it possible to install ODH in OKD? There seems to be an issue with:

  Warning  Failed          70m (x2 over 70m)    kubelet            Failed to pull image "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:4bef31eb993feb6f1096b51b4876c65a6fb1f4401fee97fa4f4542b6b7c9bc46": rpc error: code = Unknown desc = unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication

and:

  Warning  Failed          70m (x4 over 71m)    kubelet            Failed to pull image "registry.redhat.io/openshift4/ose-cli@sha256:25fef269ac6e7491cb8340119a9b473acbeb53bc6970ad029fdaae59c3d0ca61": rpc error: code = Unknown desc = unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication

Is there a workaround to make OpenDataHub work in an OKD cluster?

[israel-hdez]

What pod is generating that error?

[itmwiw]

@israel-hdez etcd, odh-dashboard and prometheus-odh-model-monitoring:

NAME                                                              READY   STATUS                  RESTARTS   AGE
data-science-pipelines-operator-controller-manager-7bbcd977gkcb   1/1     Running                 0          6h57m
etcd-cc4d875c-8xb64                                               0/1     Init:ImagePullBackOff   0          6h58m
modelmesh-controller-6856699dd5-8p4tm                             1/1     Running                 0          6h58m
modelmesh-controller-6856699dd5-k24ww                             1/1     Running                 0          6h58m
modelmesh-controller-6856699dd5-vcqt6                             1/1     Running                 0          6h58m
notebook-controller-deployment-78749747bf-hsjnh                   1/1     Running                 0          7h
odh-dashboard-5d4676fc95-mqkkl                                    1/2     ImagePullBackOff        0          7h2m
odh-dashboard-5d4676fc95-rg7xj                                    1/2     ImagePullBackOff        0          7h2m
odh-model-controller-6bd7dc4875-h24tp                             1/1     Running                 0          6h58m
odh-model-controller-6bd7dc4875-tmfr6                             1/1     Running                 0          6h58m
odh-model-controller-6bd7dc4875-vqzxd                             1/1     Running                 0          6h58m
odh-notebook-controller-manager-8676969f8b-9q6th                  1/1     Running                 0          7h
prometheus-odh-model-monitoring-0                                 2/3     ImagePullBackOff        0          6h57m
prometheus-odh-model-monitoring-1                                 2/3     ImagePullBackOff        0          6h57m
prometheus-odh-model-monitoring-2                                 2/3     ImagePullBackOff        0          6h57m
prometheus-odh-monitoring-0                                       2/2     Running                 0          7h
prometheus-odh-monitoring-1                                       2/2     Running                 0          7h
prometheus-operator-65d476c478-hfwk8                              1/1     Running                 0          7h1m

[lucferbux]

@andrewballantyne @LaVLaS
This is a fair request, we are currently using an image from the red hat registry, forcing users to be logged in. Not sure if that's the desired state but that can cause some issues.

[LaVLaS]

This is a legitimate request that would require us to ensure that every image and OpenShift feature we rely on across the ODH is "freely" accessible without Red Hat authenticated access. In certain cases we would probably need to manually build and host certain container images that only exist on the authenticated Red Hat registry

[lucferbux]

I'm gonna add this comment opendatahub-io/odh-dashboard#1779 (comment) to the conversation.
As @shalberd pointed out:

For non-corporate users, Red Hat Developer program should give easy access to registry.redhat.io via a Red Hat Login. https://access.redhat.com/RegistryAuthentication

Maybe that's a fair request to do, the alternative is just usin registry.access.redhat.com as detailed here: https://access.redhat.com/RegistryAuthentication

[shalberd]

for that image, it seems registry.access.redhat.com is not possible to use.

 docker pull registry.access.redhat.com/openshift4/ose-oauth-proxy:v4.10
Error response from daemon: unsupported: This repo requires terms acceptance and is only available on registry.redhat.io

As it says in the documentation above:
"Although both registry.access.redhat.com and registry.redhat.io hold essentially the same container images, some images require an active Red Hat account and are only available from registry.redhat.io."

I kind of wonder how you even installed OKD itself without registry.redhat.io access ... I was under the impression that you need authenticated access for it. For example, the OKD assisted installer docs also talk about the image pull secret and Red Hat Hybrid Cloud Console.

[shalberd]

@itmwiw

Is it possible to install ODH in OKD

usually, during OKD install, you add a global image pull secret to registry.redhat.io.

https://docs.okd.io/latest/openshift_images/managing_images/using-image-pull-secrets.html

In the past, I always added that global image pull secret, making the issue in other namespaces non-limiting.

Secret available after logging in to Red Hat Openshift Cluster Manager, with a Red Hat Developer Program account.

I just tried this with a private Red Hat Account at https://console.redhat.com/openshift/install/pull-secret and it gave me the image pull secret after login, even though I never formally joined any developer program.

But I did a trial install of ROSA and also was registering for access to Code Ready Containers. I think after my access to Code Ready Containers, I was linked from my Red Hat Account to Red Hat Developer program.

In any case, via a Red Hat Account and the Red Hat Developer program, users can without any issues get the image pull secret containign credentials for registry.redhat.io and registry.redhat.io and registry.connect.redhat.com, among others.

[itmwiw]

@shalberd You can install OKD without a pull secret. My install-config.yaml contains the following:

pullSecret: '{"auths":{"fake":{"auth":"aWQ6cGFzcwo="}}}

The pull secret is only required for OpenShift. As for the documentation, you may sometimes encounter 'OpenShift' references in OKD documentation that haven't been cleaned yet, such as 'SNO,' which appears in OKD docs but is exclusive to OpenShift.

If you use a pull secret from Redhat, you're abiding the ToS which has evaluation period : okd-project/okd#1237

[davidjsherman]

My understanding is that the trial period is only for RedHat's hybrid cloud console, which provides recommendations

[itmwiw]

@davidjsherman I don't think so. According to the information available here: https://access.redhat.com/RegistryAuthentication,
The two options to connect to the registry are:

Red Hat Developer Program: Signing up for a free developer account gives you access to developer tools and programs.
30-day Trial Subscription: Signing for a 30-day trial subscription gives you access to selected Red Hat software products.

The first one probably means no deployment to production and no usage as a company, while the second one has a trial period. None of those options are 'open source'. This is probably why OKD doesn't use any 'pull secret' as the community edition of OpenShift.

[itmwiw]

Hello,
Any news regarding this issue?
Thanks a lot.

[jiridanek]

Same issue still present, I hit it at

• test/e2e/standard/NotebookST.java fails to start on OKD because images from authenticated Red Hat registry are needed skodjob/odh-e2e#73

[jiridanek]

It appears that the registry.redhat.io/openshift4/ose-oauth-proxy image has an unauthenticated OKD build available at https://quay.io/repository/openshift/origin-oauth-proxy

Credits for pointing out the way go to

• [TASK] Mirror the quay.io/openshift/origin-oauth-proxy image to Longhorn repo similar to what we are doing for CSI sidecar images longhorn/longhorn#8329