How to have a CRD depend on a "healthy" service?

[kdubb]

We have implemented a number of CRDs using DependentResources to successfully manage their dependencies; it's working quite well.

Some of our CRDs need to apply their configuration to a running service; an example would be configuring a secret engine in Vault and continuously ensuring its configuration. Although we have a number of similar scenarios.

How should one go about tracking the Vault service. Obviously our CRD would add a reference to the target service. This would give us not only a target service to apply to but also provide an object to track. Unfortunately I'm having trouble figuring out how to get notified of changes in the service (e.g. going from healthy to unhealthy and vice/versa). Since Service objects don't actually carry a health status.

My thought was to add DependentResource for the Service and then use that to jump to the Pods using the selector and check if any are healthy. This is where it's not fitting together for me though.

Any help appreciated!

[csviri]

Hi @kdubb , Vault seems to have a health endpoint: https://developer.hashicorp.com/vault/api-docs/system/health

Are you deploying Vault yourself, this is there a Deployment or StatefulSet somewhere for it? You can track that.
See also: https://javaoperatorsdk.io/docs/dependent-resources#read-only-dependent-resources-vs-event-source

Another option is to poll the endpoint with a PollingEventSource (or PerResourcePollingEventSource) to have info about the health in the case what you can use during the reconciliation.

[kdubb]

Yes all our services would be Service fronted (either Deployment or StatefulSet). I have read the provided link, and we have used that strategy already with great effectiveness.

The problem I'm having is with two-step dependencies. Pointing the CRD at the Deployment or StatefulSet doesn't give us the Service (which we will need to configure our REST client). I would like to link to the Service and use that to determine Deployment or Stateful set using labels. Instead of requiring the CRD to specify both the Service and StatefulSet or Deployment.

[csviri]

Yeah, I don't see much of a problem there, so you can just can just watch resources (Thus have informer event sources for those resources, does not matter if if is read only dependent or just informer), and just search for the resource based on the label.

You can add indexes based on label in the informers:
https://github.com/java-operator-sdk/java-operator-sdk/blob/5f05c70f24f7835e279abfb82e5ae5cccd584109/operator-framework-core/src/main/java/io/javaoperatorsdk/operator/processing/event/source/informer/InformerManager.java#L191-L191

[kdubb]

Can you provide an example or link to an example of how to create an InformerEventSource that uses a selector (in their familiar form of Map<String, String>) that is taken from a secondary object?

I had seen withLabelSelector but it takes a single String and it seems like the selector would need to be on the primary object.

[csviri]

The selector is a string since fabric8 Client API provides it that way, and that is on purpose since, since selectors are not just key value pairs, see the format of the selector in k8s docs: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#label-selectors

I had seen withLabelSelector but it takes a single String and it seems like the selector would need to be on the primary object.

Not sure if I'm following this. The selector in InformerEventSource (IES) will be always on target type of IES.

You cannot have these selectors dynamically added based on other objects. But if I understand it correctly you don't want to have it.
Let's say you have a service:

apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  selector:
    app.kubernetes.io/name: MyApp
  ports:
    - protocol: TCP
      port: 80
      targetPort: 9376

That will point to the pods with the label: app.kubernetes.io/name: MyApp . So if I understand that you want to check if the target service is healthy, I'm not sure if it would not be enough to check if there are healthy pods. So just select the pods based on selector, but that not by creating an informer with the target label selector, since you can have more services you don't want to create an informer for every selector, but query the cache of the IES. (ideally have an index for labels)

[csviri]

You can add also IES for Deployments and StatefulSets and check if those are creating the pods, and are healthy, but you either don't use selector for them, or just add an additional label for those resources you are watching, independently of label selector in Service. (Just to not watch all the deployments and statuful sets on the cluster, just yours)

[csviri]

hope it is clear how do I mean this :)