ENI allocates/frees in loop with custom networking and WARM_ENI_TARGET=0

[jayanthvn]

What happened:

If custom networking is enabled and WARM_ENI_TARGET = 0, nodes which have no pods including coreDNS behind the secondary ENI will continuously free and add the secondary ENI.

Attach logs

cat /var/log/aws-routed-eni/ipamd.log | grep "Created a new"
{"level":"info","ts":"2021-04-30T03:48:00.292Z","caller":"awsutils/awsutils.go:674","msg":"Created a new ENI: eni-00403d554ade0d048"}
{"level":"info","ts":"2021-04-30T05:25:14.616Z","caller":"awsutils/awsutils.go:674","msg":"Created a new ENI: eni-07fc36da573f67a17"}
{"level":"info","ts":"2021-04-30T05:29:24.806Z","caller":"awsutils/awsutils.go:674","msg":"Created a new ENI: eni-0059aeefc485ad649"}
{"level":"info","ts":"2021-04-30T05:33:35.817Z","caller":"awsutils/awsutils.go:674","msg":"Created a new ENI: eni-00adbdaf675289a01"}
{"level":"info","ts":"2021-04-30T05:37:45.499Z","caller":"awsutils/awsutils.go:674","msg":"Created a new ENI: eni-006765cc28a46d9d2"}
{"level":"info","ts":"2021-04-30T05:41:55.402Z","caller":"awsutils/awsutils.go:674","msg":"Created a new ENI: eni-0178184612cc5184d"}
[ec2-user@ip-192-168-23-109 ~]$ cat /var/log/aws-routed-eni/ipamd.log | grep "Successfully deleted"
{"level":"info","ts":"2021-04-30T05:25:09.195Z","caller":"retry/retry.go:70","msg":"Successfully deleted ENI: eni-00403d554ade0d048"}
{"level":"info","ts":"2021-04-30T05:29:19.357Z","caller":"retry/retry.go:70","msg":"Successfully deleted ENI: eni-07fc36da573f67a17"}
{"level":"info","ts":"2021-04-30T05:33:30.377Z","caller":"retry/retry.go:70","msg":"Successfully deleted ENI: eni-0059aeefc485ad649"}
{"level":"info","ts":"2021-04-30T05:37:40.043Z","caller":"retry/retry.go:70","msg":"Successfully deleted ENI: eni-00adbdaf675289a01"}
{"level":"info","ts":"2021-04-30T05:41:49.911Z","caller":"retry/retry.go:70","msg":"Successfully deleted ENI: eni-006765cc28a46d9d2"}
{"level":"info","ts":"2021-04-30T05:46:00.768Z","caller":"retry/retry.go:70","msg":"Successfully deleted ENI: eni-0178184612cc5184d"}

Issue is not seen without custom networking since the primary ENI is not freed.

What you expected to happen:
ENI should not be freed and one ENI should be reserved for custom networking scenarios.

How to reproduce it (as minimally and precisely as possible):
Have 2 nodes, enabled custom networking and set WARM_ENI_TARGET=0

Anything else we need to know?:
Better approach is to not rely on counters, walk through the ENIs and cache available ENI. Compare that with WARM_ENI_TARGET or if custom networking is enabled.

Environment:

• Kubernetes version (use kubectl version):
• CNI Version -> Day 1 issue
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 14 days

[jayanthvn]

PR is pending review. /not stale

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 14 days

[jayanthvn]

/not stale

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 14 days

[github-actions]

Issue closed due to inactivity.

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 14 days

[jdn5126]

The current plan for addressing the scenario where WARM_ENI_TARGET=0 and all ENIs are full is to enhance the CNI/IPAMD interface to attach ENIs on demand. The current recommendation is to NOT use WARM_ENI_TARGET=0, but to instead set WARM_IP_TARGET=1 and a MAX_ENI value for your instance.

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 14 days

[jdn5126]

/not stale

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 14 days

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 14 days

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 14 days

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 14 days

[github-actions]

Issue closed due to inactivity.

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 14 days