[Bug]: Issue with failed webhook after upgrade to v2.4 from v2.2

[ssiegel95]

ODH Component

ODH Notebook Controller

Current Behavior

After upgrading our ODH operator version to 2.4 from 2.2, attempts to create new workbenches fail with the following error:

Internal error occurred: failed calling webhook "notebooks.opendatahub.io": failed to call webhook: Post "https://odh-notebook-controller-webhook-service.opendatahub.svc:443/mutate-notebook-v1?timeout=10s": EOF. 

Expected Behavior

New workbenches are able to be created.

Steps To Reproduce

No response

Workaround (if any)

No response

What browsers are you seeing the problem on? (If applicable)

Chrome

Open Data Hub Version

No response

Anything else

OpenShift version
4.14.5

Node Feature Discovery Operator
4.14.0-202312052033 provided by Red Hat

[krisan]

Also in ODH 2.7.0 (new install), OpenShift 4.14.8:

request failed:

POST https://odh-dashboard-opendatahub.host/api/k8s/apis/kubeflow.org/v1/namespaces/test-project/notebooks

Response

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": 'Internal error occurred: failed calling webhook "notebooks.opendatahub.io": failed to call webhook: Post "[https://odh-notebook-controller-webhook-service.opendatahub.svc:443/mutate-notebook-v1?timeout=10s](https://odh-notebook-controller-webhook-service.opendatahub.svc/mutate-notebook-v1?timeout=10s)": http: server gave HTTP response to HTTPS client',
  "reason": "InternalError",
  "details":
    {
      "causes":
        [
          {
            "message": 'failed calling webhook "notebooks.opendatahub.io": failed to call webhook: Post "[https://odh-notebook-controller-webhook-service.opendatahub.svc:443/mutate-notebook-v1?timeout=10s](https://odh-notebook-controller-webhook-service.opendatahub.svc/mutate-notebook-v1?timeout=10s)": http: server gave HTTP response to HTTPS client',
          },
        ],
    },
  "code": 500,
}

[lucferbux]

@Jooho @Xaenalt I've seen this a few times in our dev clusters, could you help us out here?

[krisan]

The same? issue on Applications/Notebook Server tab - on notebook creation:

OpenShift 4.14.8 (master), workers upgraded to OpenShift 4.14.11.

# Request
# (also for PATCH)
POST /api/notebooks HTTP/1.1
...
Accept: application/json, text/plain, */*
Content-Type: application/json
...

# Response
HTTP/1.1 400 Bad Request
content-type: text/plain; charset=utf-8
...
Internal error occurred: failed calling webhook "notebooks.opendatahub.io": failed to call webhook: Post "[https://odh-notebook-controller-webhook-service.opendatahub.svc:443/mutate-notebook-v1?timeout=10s](https://odh-notebook-controller-webhook-service.opendatahub.svc/mutate-notebook-v1?timeout=10s)": http: server gave HTTP response to HTTPS client

[krisan]

Working workaround in my case - downgrade to last working version:

• remove DSC, DSCI, uninstall ODH operator, remove opendatahub namespace
• install ODH 2.2.1 (set manual)

On ODH 2.3.0 the same issue

[zdtsw]

is this still a valid bug? i think better upgrade to the latest ODH 2.16 and see how it goes

[Santimoso]

Tested ODH 2.16 and 2.17 with Openshift 4.14 and faced same issue...

Tomorrow I will try to clean the environment and test version 2.2.1 as suggested... but it's true that 2.2.1 was released in Oct 10, 2023 (1 year ago).

If anyone already solved this problem in newer versions, can you share your workarround?