I'm surprised this is happening. By default CRs are pre-deployed. This behavior can be turned off by an annotation on the CRD https://github.com/Shopify/krane#customizing-behaviour-with-annotations krane.shopify.io/predeploye: false .

Can you add the example CR and CRD to a comment in this issue?

We're setting up calico as a separate workload in every cluster. Whenever we deploy a different workload to a cluster, calico is already there and the krane discovery is going to find a CRD NetworkPolicy resource coming from it. That overrides the predeployment for the CR NetworkPolicy for any workload we're deploying.

That happens in every workload we deploy to the cluster, no matter what the CR NetworkPolicy is, because the hash for CRDs comes after the hash for CRs when concatenating in DeployTask.predeploysequence (https://github.com/Shopify/krane/blob/master/lib/krane/deploy_task.rb#L59-L78), specifically:

Hash[before_crs + crs + after_crs]

Examples

1. CRD NetworkPolicy
   We're using calico CRDs, e.g.:

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: globalnetworkpolicies.crd.projectcalico.org
  labels:
    workload: calico
spec:
  scope: Cluster
  group: crd.projectcalico.org
  version: v1
  names:
    kind: GlobalNetworkPolicy
    plural: globalnetworkpolicies
    singular: globalnetworkpolicy

2. CR NetworkPolicy
   Any CR NetworkPolicy will have this issue, a minimal example:

---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: example-network-policy
spec:
  podSelector: {}
  ingress:
    - {}
  policyTypes:
    - Ingress

Oh, I see NetworkPolicy exists in extensions / networking.k8s.io depending on k8s version.

When you use Calico it also adds a NetworkPolicy CRD and because of the way we construct the hash: https://github.com/Shopify/krane/blob/master/lib/krane/deploy_task.rb#L77 we only consider Calico's NetworkPolicy as something to predeploy. Yep this is 100% a bug.

So fixing this is a bit tricky. My two stabs at it are in #774, but neither are quite right. I've got a 3rd idea (we have to track the groups to pre-deploy and the groups not to pre-deploy), but might not finish it today.

Yay, glad it makes sense with more context 🎉

I couldn't think of an elegant way to solve it, because in some cases you may want to predeploy CRDs and not CRs (seemed like that was the case for the feature request that the grouping was addressing, i.e. #728).

I was wondering if there is a definite set of cases for which resources you may or may not want to predeploy, or can it depend on the workload/how the cluster is set up?

I was wondering if there is a definite set of cases for which resources you may or may not want to predeploy, or can it depend on the workload/how the cluster is set up?

Its up to each operator if they want to pre-deploy CRs or not. As an example: If you use a CR to do canary deploys, you wouldn't want that to be pre-deployed.

Hi @dturn, happy new year 🎉

Did you end up making any progress on a different approach? Anything I could do to help?

Happy to provide more info and brainstorm :)

Following up @dturn :) Anything I could do to help?

I'm slowly making progress on #774 but when I went to write the test I realized this is going to take longer than expected. It turns out there are at least 3 places we assume Kinds are unique: predeploy, KubernetesResource#build, and ResourceCache.

The PR originally was trying to solve the first issue. Solving the second one is done locally, but still haven't had enough time to finish up the 3rd. Assuming no more surprises, I'm optimistic I'll get it done this week.

@trueskawka you've probably figured this out, but I haven't had as much time to work on this as needed. I can't commit to getting this fixed in a reasonable time frame. If you're interested in taking over let me know and I'll do my best to support you. Otherwise, you can follow this this issue and I'll keep it updated.

Hi @dturn!

We were able to work around it by doing our networking a bit differently and not having to depend on calico as much. I'm happy to try and PR a change to fix it for others who might be still running into this issue.

Gonna do a PR sometime in the next two weeks and @-mention you :)

@dturn @trueskawka Has anything changed since the last comments here? I'm running into this same problem.

@dturn I know you stated there's three bugs to fix, but I wonder if it's necessary to fix them all together. If you have a solution for the specific issue described here, is it not possible to ship that on its own, acknowledging that other similar issues will remain?