Honor publishNotReadyAddresses flag for non headless services

[gk-fschubert]

What happened:
normal services which have the flag "publishNotReadyAddresses: true" defined does not get reflected on the exported services.

What you expected to happen:
The settings defined for a service which I want to export should get reflected on exported service.

It seems to be that this has been done for headless services: #2646
But not for non-headless services. Or a least the configuration item is missing in the exported service.

How to reproduce it (as minimally and precisely as possible):

1. Deployment of nginx with a readiness probe which never get's true

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: nginx-deployment
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - image: nginx:1.14.2
          imagePullPolicy: IfNotPresent
          name: nginx
          ports:
            - containerPort: 80
              protocol: TCP
          readinessProbe:
            exec:
              command:
                - /bin/bash
                - -ec
                - asdkjfhasdfkj

2. Service with publishNotReadyAddresses=true

apiVersion: v1
kind: Service
metadata:
  name: nginx
  namespace: default
spec:
  publishNotReadyAddresses: true
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80

3. export the service with subclt

subctl  export service nginx -n default  

Result:
Exported service does not get the flag publishNotReadyAddresses: true

Environment:
subctl version: v0.18.0

[tpantelis]

This flag only applies to non headless services

Exported service does not get the flag publishNotReadyAddresses: true

What behavior are you expecting wrt exported service?

[gk-fschubert]

This flag only applies to non headless services

It should apply for both or do I miss something?

Exported service does not get the flag publishNotReadyAddresses: true

What behavior are you expecting wrt exported service?

The exported service need the same flag as the service which submariner should export in order to have a consistent behaviour.

[tpantelis]

This flag only applies to non headless services

It should apply for both or do I miss something?

Actually I meant to say it "only applies to headless services"

Per the definition of the publishNotReadyAddresses flag in the K8s docs:

The primary use case for setting this field is for a StatefulSet's Headless Service to propagate SRV DNS records for its Pods for the purpose of peer discovery.

So for headless services, we have a use case for this flag and we implemented it.

For ClusterIP (ie non-headless) services we don't publish the local pod endpoint IPs to other clusters in the EndpointSlices - we only publish the service's cluster IP along with the ready state determined from the readiness of the underlying pod endpoints. So we are already technically publishing not-ready-addresses for ClusterIP services even though the semantics of the publishNotReadyAddresses flag doesn't really apply.

Exported service does not get the flag publishNotReadyAddresses: true

What behavior are you expecting wrt exported service?

The exported service need the same flag as the service which submariner should export in order to have a consistent behaviour.

I'm not following here. The publishNotReadyAddresses flag is on the Service resource. We don't actually export a Service resource instance - we export a ServiceImport along with EndpointSlices. So, for ClusterIP services, where are you expecting the flag to be propagated and what behavior are you expecting? Or, to rephrase, what behavior is missing or what problem did you observe that led you to report this issue?

[gk-fschubert]

I'm not following here. The publishNotReadyAddresses flag is on the Service resource. We don't actually export a Service resource instance - we export a ServiceImport along with EndpointSlices. So, for ClusterIP services, where are you expecting the flag to be propagated and what behavior are you expecting?

Ok then may I don't know the exact component which is causing issues here.
Yes technically the IP address is published before the pod is ready. But the service will not be able to forward traffic if there is no ready pod in it.

What I mean is:
When I create a clusterIP service with "publishNotReadyAddresses: true" and then export that with subctl a new service get's created.
That new service e.g. "submariner-lpq6zr4tl4o5qvrv2t7o3l3gawkagast" does not have the value "publishNotReadyAddresses" at all and thus is not forwarding traffic to a non-ready pod.

Or, to rephrase, what behavior is missing or what problem did you observe that led you to report this issue?

I want to run a Cassandra database across multiple k8s clusters and because they have overlapping CIDR networks, Globalnet is used.
Each Cassandra rack (only one server per rack) get's a service which will exported over submariner which they then use as broadcast address.
And each rack have to be reachable before the pod is ready.

[tpantelis]

That new service e.g. "submariner-lpq6zr4tl4o5qvrv2t7o3l3gawkagast" does not have the value "publishNotReadyAddresses" at all and thus is not forwarding traffic to a non-ready pod.

Ok - that's an internal local service created by Globalnet, I believe to handle ingress traffic for the global IP and route to the exported service's pods @aswinsuryan. It's not actually exported to other clusters - I was confused by the terminology. We can certainly propagate the publishNotReadyAddresses flag from the exported service here. Have/can you verify that setting this flag on the internal service addresses the problem you're seeing?

[gk-fschubert]

I was confused by the terminology

Sorry!

Have/can you verify that setting this flag on the internal service addresses the problem you're seeing?

Yes. But it get's automatically reverted