How to setup Dependent resource based on CRD which is not present when controller start

[dmvolod]

Hello, team!

We have a situation, when our operator can have an optional dependent (second) custom resources based on CRD's which are not present when controller start. For example it can be some of the servicemonitors.monitoring.coreos.com CRD, which can be installed in the Kubernetes cluster after our operator has been installed.

I know, that this case can be handled by validation for CRD is present or not with client.getApiResources and create/update/delete CR based on this kind of dynamic CRD manually.
But would be nice to have out-of-box solution for that, as if define this Dependent resource manually without additional behavior, InformerEventSource will try to initialize it's cache with following error

2023-03-06 11:01:16,014 ERROR [io.qua.run.Application] (Quarkus Main Thread) Failed to start application (with profile [dev]): io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://127.0.0.1:37597/apis/monitoring.coreos.com/v1/servicemonitors. Message: Not Found.
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.requestFailure(OperationSupport.java:742)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.requestFailure(OperationSupport.java:722)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.assertResponseCode(OperationSupport.java:673)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.lambda$handleResponse$0(OperationSupport.java:601)
	at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:646)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.lambda$retryWithExponentialBackoff$2(OperationSupport.java:643)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
	at io.fabric8.kubernetes.client.http.ByteArrayBodyHandler.onBodyDone(ByteArrayBodyHandler.java:52)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
	at io.fabric8.kubernetes.client.okhttp.OkHttpClientImpl$OkHttpAsyncBody.doConsume(OkHttpClientImpl.java:133)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)

Another option, we can create SharedIndexInformer manually, but can't find solution how to invoke reconcile loop from the ResourceEventHandler implementation.

Thanks in advance.

[metacosm]

Hi @dmvolod, how would you envision this working? Do you declare your dependents declaratively (i.e. using annotations)? Which version of the SDK / Quarkus extension are you using?

[dmvolod]

Hi @metacosm

Thanks for response. Tried both options: declare as @Dependent annotation or override prepareEventSources in Reconciler implementaion with manual code like this

var pipelinesInformerConfig = InformerConfiguration.from(ServiceMonitor.class, context)
                .withLabelSelector(Constants.K8S_OPERATOR_MANAGED_BY_SELECTOR)
                .build();
        return EventSourceInitializer.nameEventSources(new InformerEventSource<>(pipelinesInformerConfig, context));

Using Quarkiverse operator SDK 5.1.1

[metacosm]

Maybe you could add a reconcilePrecondition to your dependent that would check that the CRD is indeed deployed before it's reconciled? That said, that might still not work because the SDK would still attempt to create the associated informer… Have you tried this?

[dmvolod]

Also, I don't have idea, instead of use new KubernetesClientBuilder().build() in the constructor how to get access to the Kubernetes client in the reconcilePrecondition implementation.

[csviri]

This is actually a valid use case to add client (make it accessible) into the context. This is not needed for resources, since those should be read from the cache, will create an issue to make it accessible.

[dmvolod]

Thanks a lot for this. Will be very helpful.

[csviri]

Added issue: #1799 will pick it up soon

[dmvolod]

Yes, tried reconcilePrecondition without success in our case.
But seems to find a solution, with stopOnInformerErrorDuringStartup disabled. Otherwise would be nice to have an option to validate CRD available via client.getApiResources and than run informer in dynamic.
In this case, following error should be suppressed

2023-03-06 21:19:00,035 WARN  [io.fab.kub.cli.inf.imp.cac.Reflector] (OkHttp Dispatcher) listSyncAndWatch failed for monitoring.coreos.com/v1/servicemonitors, will retry: java.util.concurrent.CompletionException: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://127.0.0.1:44809/apis/monitoring.coreos.com/v1/servicemonitors. Message: Not Found.
	at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315)
	at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320)
	at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:649)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.lambda$retryWithExponentialBackoff$2(OperationSupport.java:643)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
	at io.fabric8.kubernetes.client.http.ByteArrayBodyHandler.onBodyDone(ByteArrayBodyHandler.java:52)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
	at io.fabric8.kubernetes.client.okhttp.OkHttpClientImpl$OkHttpAsyncBody.doConsume(OkHttpClientImpl.java:133)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)

Another option, it's possible to create informer manually and than, have to invoke a reconcile() from the ResourceEventHandler methods. Is it possible to do this?

[csviri]

(Just a side note: TBH it's just a very little margin that you win in that case, since stopping the controllers evicts the caches, and the startup time usually takes long because all the caches of event sources are filled before the first reconciliation happens. )

[dmvolod]

Yes, I now, but the single case I would like to avoid, that the operator Pod will not restart.For example from monitoring point of view this operation will be transparent.

[dmvolod]

Also for dev environment this approach will not work out-of-the-box

[metacosm]

Can you please open an issue in the extension project?

[dmvolod]

Yes, sure. Will do, thanks.