fail to use make deploy on local kind cluster for development

[SamYuan1990]

Hi team,

ref to https://sdk.operatorframework.io/docs/building-operators/golang/tutorial/#2-run-as-a-deployment-inside-the-cluster
I am plan to use make deploy to deploy some local change on a kind cluster (started with ./network kind)
but an error occurs as

The Deployment "operator-controller-manager" is invalid: spec.template.metadata.labels: Invalid value: map[string]string{"control-plane":"controller-manager"}: `selector` does not match template `labels`

full steps and logs

yuanyi@yuanyideMacBook-Pro sample-network % ./network kind
Initializing kind cluster "kind":
✅ - Creating cluster "kind" ...
✅ - Launching container registry "kind-registry" at localhost:5000 ...
🏁 - Cluster is ready.
yuanyi@yuanyideMacBook-Pro sample-network % cd ..
yuanyi@yuanyideMacBook-Pro fabric-operator % make deploy
/usr/local/bin/controller-gen "crd:crdVersions=v1" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
cd config/manager && /usr/local/bin/kustomize edit set image controller=controller:latest
/usr/local/bin/kustomize build config/default | kubectl apply -f -
namespace/operator-system created
customresourcedefinition.apiextensions.k8s.io/ibpcas.ibp.com created
customresourcedefinition.apiextensions.k8s.io/ibpconsoles.ibp.com created
customresourcedefinition.apiextensions.k8s.io/ibporderers.ibp.com created
customresourcedefinition.apiextensions.k8s.io/ibppeers.ibp.com created
serviceaccount/operator-controller-manager created
role.rbac.authorization.k8s.io/operator-leader-election-role created
clusterrole.rbac.authorization.k8s.io/operator-manager-role created
clusterrole.rbac.authorization.k8s.io/operator-metrics-reader created
clusterrole.rbac.authorization.k8s.io/operator-proxy-role created
rolebinding.rbac.authorization.k8s.io/operator-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/operator-operator created
clusterrolebinding.rbac.authorization.k8s.io/operator-proxy-rolebinding created
service/operator-controller-manager-metrics-service created
The Deployment "operator-controller-manager" is invalid: spec.template.metadata.labels: Invalid value: map[string]string{"control-plane":"controller-manager"}: `selector` does not match template `labels`
make: *** [deploy] Error 1
yuanyi@yuanyideMacBook-Pro fabric-operator % 

please advise

[jkneubuh]

Hi @SamYuan1990

There is limited support for the Makefile, operator, CRDs, and a local cluster in the current project. You are breaking new ground in this area, and we welcome improvements in the way that the local development occurs with the operator. Please make sure to update any instructions in the project README / CONTRIBUTING / etc. files as you discover new practices for making it easier to build, test, and run in a local sandbox.

The current makefiles know how to create a KIND cluster (make kind), build CRDs (make manifests), and install CRDs to the local kube context with (make install). In these environments, the assumption is that the golang operator will run directly on the host system, connecting to the kube API controller as specified in the user's kube context or ~/.kube/config. Note that this is a little different than running from within the network shell scripts, which launch the operator in a service account and require additional RBAC settings for correct operation.

Also note that there are still some template files created by kubebuilder / operator-sdk, which to my knowledge are NOT used by the project. I agree these are confusing and we should either make them work correctly, or remove the targets as they are confusing for new developers.

Currently there is NO make target that (correctly) launches the operator as a local binary, running against the current kube config settings. Also there is NO makefile target that correctly launches the operator as a deployment.

The make kind cluster IS used for creating a local cluster that can be used to run integration test suites with ginkgo, and for running the operator binary locally (e.g. in a step debugger).

[SamYuan1990]

Hi @SamYuan1990

There is limited support for the Makefile, operator, CRDs, and a local cluster in the current project. You are breaking new ground in this area, and we welcome improvements in the way that the local development occurs with the operator. Please make sure to update any instructions in the project README / CONTRIBUTING / etc. files as you discover new practices for making it easier to build, test, and run in a local sandbox.

The current makefiles know how to create a KIND cluster (make kind), build CRDs (make manifests), and install CRDs to the local kube context with (make install). In these environments, the assumption is that the golang operator will run directly on the host system, connecting to the kube API controller as specified in the user's kube context or ~/.kube/config. Note that this is a little different than running from within the network shell scripts, which launch the operator in a service account and require additional RBAC settings for correct operation.

Also note that there are still some template files created by kubebuilder / operator-sdk, which to my knowledge are NOT used by the project. I agree these are confusing and we should either make them work correctly, or remove the targets as they are confusing for new developers.

Currently there is NO make target that (correctly) launches the operator as a local binary, running against the current kube config settings. Also there is NO makefile target that correctly launches the operator as a deployment.

The make kind cluster IS used for creating a local cluster that can be used to run integration test suites with ginkgo, and for running the operator binary locally (e.g. in a step debugger).

ok, I will try with make kind the only thing is that I hope make kind able to support local image upload/use local registory.

[SamYuan1990]

Hi @SamYuan1990

There is limited support for the Makefile, operator, CRDs, and a local cluster in the current project. You are breaking new ground in this area, and we welcome improvements in the way that the local development occurs with the operator. Please make sure to update any instructions in the project README / CONTRIBUTING / etc. files as you discover new practices for making it easier to build, test, and run in a local sandbox.

The current makefiles know how to create a KIND cluster (make kind), build CRDs (make manifests), and install CRDs to the local kube context with (make install). In these environments, the assumption is that the golang operator will run directly on the host system, connecting to the kube API controller as specified in the user's kube context or ~/.kube/config. Note that this is a little different than running from within the network shell scripts, which launch the operator in a service account and require additional RBAC settings for correct operation.

Also note that there are still some template files created by kubebuilder / operator-sdk, which to my knowledge are NOT used by the project. I agree these are confusing and we should either make them work correctly, or remove the targets as they are confusing for new developers.

Currently there is NO make target that (correctly) launches the operator as a local binary, running against the current kube config settings. Also there is NO makefile target that correctly launches the operator as a deployment.

The make kind cluster IS used for creating a local cluster that can be used to run integration test suites with ginkgo, and for running the operator binary locally (e.g. in a step debugger).

hi @jkneubuh ,

https://github.com/kubernetes/ingress-nginx.git/deploy/static/provider/kind?ref=controller-v1.1.2 is page not found, and it is used here https://github.com/hyperledger-labs/fabric-operator/blob/main/config/ingress/kind/kustomization.yaml#L22

so could you please share the logic for kind base cluster should have to support development?

[SamYuan1990]

or you can help me figure out which is the correct file ? https://github.com/kubernetes/ingress-nginx/tree/main/deploy/static/provider/kind

[jkneubuh]

Hi Sam. Yes, that is the correct file / URL for the nginx ingress.

The URL used by kustomization will not resolve correctly in the browser. This is a feature used by kustomize (or kubectl -k) which will use the Hashicorp go-getter library to resolve URLs from various sources, including git ssh, s3, http, ftp, etc. etc.

This is a separate project, but @mbwhite has made some significant progress on setting up Fabric networks using Ansible playbooks and fabric-operator. In that environment, the KIND + nginx + coreDNS updates (localho.st -> Ingress IP) are treated as required dependencies for the operator and Ansible plays.

Take a look at full-stack-asset-transfer to see a "recipe" for creating a KIND cluster that is ready to go for the operator. I think these are the steps that need be carried over into the operator Makefile. The Ansible project example uses just, which is a very close approximation of a Makefile.

[jkneubuh]

For the localhost:5000 registry. .... do we want or need to start a local docker container registry? Can we accomplish this with a kind load docker-image .... command and the use of imagePullPolicy=IfNotPresent in the fabric-operator Deployment?