CNI delegation failed due to error:Error delegating ADD to CNI plugin:sriov because:OS exec call faild:netplugin failed with no error message

[nknkgithub]

Is this a BUG REPORT or FEATURE REQUEST?:

bug

What happened:

We have sriov-o1c-host0 cluster network and 4 pods are using that CNI.
1st pod { "clusterNetwork": "sriov-o1c-host0", "ip6": "fdfb:6442:1:673::1", "proutes6": {"::0/0": "fdfb:6442:1:673::7fff"} },
2nd pod { "clusterNetwork": "sriov-o1c-host0", "ip6": "fdfb:6442:1:673::2", "proutes6": {"::0/0": "fdfb:6442:1:673::7fff"} },
3rd pod { "clusterNetwork": "sriov-o1c-host0", "ip": "10.0.106.65", "proutes": {"11.0.0.0/20": "10.0.106.72"} }
4th pod {"clusterNetwork": "sriov-o1c-host0", "ip": "10.0.106.72", "ip6": "fdfb:6442:1:673::7fff" },

But when we deploy pods one pod is in container creating state , and when pod is described below error is observed.

Warning FailedCreatePodSandBox 30m kubelet, controller-0 Failed to create pod sandbox: rpc error: code = Unknown desc = faled to setup network for sandbox "cd96d7ad7726d3ff73b4779eb98344eae19c95791cc26583652c5709922af75e": CNI network could not be set up: CNI operaton for network:sriov-o1c-host0 failed with:CNI delegation failed due to error:Error delegating ADD to CNI plugin:sriov because:OS exec call faild:netplugin failed with no error message

What you expected to happen:

All the pods should be up and running

How to reproduce it:

Deploy pods with cluster network as shown above. One pod will not come up.
This occurs sometimes

Anything else we need to know?:

Danm cleaner pods are running in the setup

kubectl get pods -A | grep -i danm-cleaner
kube-system               danm-cleaner-5dtgr                                                1/1     Running     0          99m

Environment:

• DANM version (use danm -version):

 /usr/libexec/cni/danm --version
2021/11/24 05:33:55 DANM binary was built from release: v4.2.1
2021/11/24 05:33:55 DANM binary was built from commit: abd3c48d_dirty

• Kubernetes version (use kubectl version):

kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.1", GitCommit:"7879fc12a63337efff607952a323df90cdc7a335", GitTreeState:"archive", BuildDate:"2021-06-10T00:00:03Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.1", GitCommit:"7879fc12a63337efff607952a323df90cdc7a335", GitTreeState:"clean", BuildDate:"2020-04-08T17:30:47Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

• DANM configuration (K8s manifests, kubeconfig files, CNI config file):

 cat /etc/cni/net.d/00-danm.conf
{
  "cniVersion": "0.3.1",
  "name": "danm_meta_cni",
  "type": "danm",
  "kubeconfig": "/etc/cni/net.d/danm-kubeconfig",
  "cniDir": "/etc/cni/net.d",
  "namingScheme": "legacy"
}

cat /etc/cni/net.d/danm-kubeconfig

---
apiVersion: v1
kind: Config
current-context: default
clusters:
  - cluster:
      certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM5akNDQWQ2Z0F3SUJBZ0lJbnA4c0grdHJsemt3RFFZSktvWklodmNOQVFFTEJRQXdJVEVmTUIwR0ExVUUKQXd3V1kyOXRiWE5qYjNCbFlteHlhWEJ6WldOaFpHMXBiakFlRncweU1UQTNNRE14TURVek5UZGFGdzAwTVRBMwpNRE14TURVek5UZGFNQ0V4SHpBZEJnTlZCQU1NRm1OdmJXMXpZMjl3WldKc2NtbHdjMlZqWVdSdGFXNHdnZ0VpCk1BMEdDU3FHU0liM0RRRUJBUVVBQTRJQkR3QXdnZ0VLQW9JQkFRQ3NIU0FWMURSTUFnZ1FtTng1TU5vL0c5V3YKR3hqa2luN0NWOHpYdVV6TFZtRjdjMG5KdjlVaDR1WG1LdzNFR3V4MnQveGx3NlhTN3c3TjdFSzcxNzF5Wm1nVwpXc3RPMWgzeDBDWVhoQ1VxOXVXNUdET2lZVnlxMUljOUZSSUQ0U2N6Z1FNb1hFT0ZhVDhlY1U0MEJNNTJqYmlnCkphb2lDZ0RVYmtNT3dMOWpTdlgzQlhpNjNybVRiZTlRMkM0dmxMUmdQK1lFRnpPaDJEYTU4cm5EdVJqVFUyaWsKOUIvNnVzeXMzZ0VFZkhxZ3R4NGNTQ0RVaFpxNVRjYTZjM2hUYVV6NU9HV3NNN2c1bEFDVDlNQ0tOOEN3R1ZNTwpBVkt2WkhPaDRYTWZlU0x3RjZHdkNBN1V6QzhYYjM3TU04REtVbjVQYnJyVUwvNUdrb1djNzI1R3QrMVJBZ01CCkFBR2pNakF3TUE4R0ExVWRFd0VCL3dRRk1BTUJBZjh3SFFZRFZSME9CQllFRkc0ZGg3bTZWcXd1bnZlN1ZKV28KcXJ1dFRmeWFNQTBHQ1NxR1NJYjNEUUVCQ3dVQUE0SUJBUUNXNjlsNlFUMjkwUkl0cXJuRldvZ05SV3dYQTlKbgpWSndCK2JsUmhoLzd2ODlTazZvWFhJZlZob0xGaDlZSEtYcFJaREd3RkVBWERTbFhUWFM1YlB6T3F2dkVmR0dCCm81TGRnZE1PR2VYMVg0OHB2TjNlUmZrRE54MFFEKzRFeVhBaVFobmMva0VJcDV0TTNMS1VEOUp5b2lCek5QL0wKSndZUGxiMkxHWFlQY0ZrRWMwTzJ6ZkZtUEtGL294ZFE0dWtDQU8zbVlFT1BEZEl6ZEI4dEYvdFBqUzQ0aWdlegpSUkRtN1RpR0wwd1VYZHQ1NlB6aTJyVEpwa3dqVHlZWTBscWNWZnBYcWkxc0R0QXBzRTRQTlZZZTVkTFdHYjRnCnpTTDRJSGtQejRMNnZnQUxKTVlWclVzNFBzZTVjZW9Gdm9vbHV6YjQ3SXpHUFljN3FiU09lNGdaCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0=
      server: https://10.96.0.1:443
    name: kubernetes
contexts:
  - context:
      cluster: kubernetes
      user: danm
    name: default
users:
  - name: danm
    user:
      token: eyJhbGciOiJSUzI1NiIsImtpZCI6IklseWJIYlBYQlJjOEJ1dHZ0Z3lOd3BLVkp1c3VJUEd3bkVRYm9aUWpFS1UifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJkYW5tLXRva2VuLXE2ZjZoIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImRhbm0iLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiIzN2YzYzg1ZS03ZDlkLTQ2NTgtOGU4YS0yOGJmNmMwNjgyYTgiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZS1zeXN0ZW06ZGFubSJ9.UH1RlL8H3a9as0fTCPQNVU84MwDqI7QbORV8hQyj7EawBB8xi3EZWHUzWaf2K6TaIXZebWc_9x2odZDIrPQucVuQqBoUmTsiI083bPwbFxpita4_N_sjdHC-VwWWyDw7KZpfSWhapLRAMTTRzb_s_VO1-VP9Pv7wGj6kCB50CsnYn-FDQf2d-uPEjjNJlhtlSU9ggwWiuceRylW_kk0n554uAJkexCooW8gfsfv2qQWOiAR9cw7oDSstBcvxDUMGW1iTkpy2b1xHP_LRrXfi6XiJRE91kk4OApB83XaPm04DNcnNnCq0BC8qKjBx1NJmOZyqKHw_qoVbxU1GHQM1jg
preferences: {}

• OS (e.g. from /etc/os-release):

cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

• Kernel (e.g. uname -a):

uname -a
Linux controller-0 3.10.0-1160.15.2.rt56.1152.el7.tis.4.x86_64 #1 SMP PREEMPT RT Wed Jun 9 20:40:45 EDT 2021 x86_64 x86_64 x86_64 GNU/Linux

• Others:

[nknkgithub]

Help required

[nknkgithub]

Any updates regarding this? Want to know cause for the issue