Multi-nic, Hairpin LB and One-click Installation

v1.1.0 -- 2020/04/07

In this version, we refactor IPAM to separate IP allocation logical from OVN. On top of that we provide a general cluster wide IPAM utility for other CNI plugins. Now other CNI plugins like macvlan/host-device/vlan etc can take advantage of subnet and static ip allocation functions in Kube-OVN.Please check this document to see how we combine Kube-OVN and Multus-CNI to provide multi-nic container network.

This version also update OVN to 20.03 and OVS to 2.13 with some x86 related optimization, which increase the control plan data plan performance and we finally support hairpin LB now.

This version also simplify the installation steps by merge all components into one image and add a new one-click installation script to help users setup a high-available, production-ready Kube-OVN container network. Please look the installation guide

IPAM

• Separate IPAM logical form OVN
• Add support for Multus-CNI

Performance

• Recycle address if pod is in failed or succeeded phase
• Delete chassis form ovn-sb when node deleted
• Only enqueue updatePod when needed
• Add x86 optimization CFLAGS
• Add support to disable encapsulation checksum

Monitor

• Diagnose will check Kube-OVN components status
• Diagnose will check crd status
• Diagnose will check kube-proxy and coredns status

Bugfix

• Use uuid to fetch lb vips
• Add inactivity_probe back
• Update svc might remove other svc that with same prefix
• IP prefix might be empty
• Enqueue subnet update to add route
• Add iptables to accept container traffic

Chore

• Update OVN to 20.03 and OVS to 2.13
• Add support for Kubernetes 1.17
• Put all component in one image to reduce distribute burden
• Add scripts to build ovs
• Add one script installer
• Add uninstall script
• Add more e2e tests

Bugfix

v1.0.1 -- 2020/03/31

This release fix bugs found in v1.0.0

Bugfix

• Use uuid to fetch lb vips
• Add inactivity_probe back
• Update svc might remove other svc that with same prefix
• IP prefix might be empty
• Enqueue subnet update to add route

v1.0.0 -- Kube-OVN GA!

Kube-OVN has evolved a year from the first release and the core function set is stable with lots of tests and community feedback.

It's time to run Kube-OVN in production!

Performance

• Disable ovn-nb inactivity_probe to enhance ovn-nbctl daemon performance
• Config ovn-openflow-probe-interval to prevent disconnection when cluster is large
• Pick ovn upstream patch to enhance ovn-controller performance

Monitoring

• Display controller logs in kubectl-ko diagnose
• Expose cni operation metrics
• Pinger check portbindings between local ovs and ovs-sb
• Pinger add timeout for dns/ovs/ovn check

Mics

• Add e2e test framework
• Move all components to kube-system namespace to use a higher priorityClass
• Refactor code for better readability

Bugfix

• If cidr block not ends with zero, reformat the cidr block
• CniServer will resync iptables to avoid manually or other software change the iptable
• Do not return not found error when first add node
• Restart ovn-nbctl daemon when it hangs
• RunGateway will restart in case init failed.
• When subnet cidr conflict requeue the subnet
• Recompute ovn-controller periodically to avoid inconsistency
• Wait for flow installed before cni return
• Add back missing lsp gc
• Delete lb if it has no backends

v0.10.2 -- Bugfix

This release fix bugs found in v0.10.1.

If you previously used v0.10.1 you can update the image tag of kube-ovn-controller, kube-ovn-cni, ovn-ovs to v0.10.2 and delete daemonset pods manually to update to the new release.

Bugfix

• restart nbctl-daemon if not response
• runGateway may fail when init
• requeue subnet add event when conflict with exist subnet
• periodically recompute ovn-controller to avoid inconsistency

v0.10.1 -- Bugfix

This release fix bugs found in v0.10.0.

If you previously used v0.10.0 you can update the image tag of kube-ovn-controller, kube-ovn-cni and kube-ovn-pinger to v0.10.1 and delete daemonset pods manually to update to the new release.

Bugfix

• If cidr block not ends with zero, reformat it to avoid add route failure
• Resync iptables to prevent rules deleted by other software
• Set ovn-openflow-probe-interval to prevent ovn-controller consumes all cpu
• Do not return not found error when first add node

Monitoring

• pinger checks port bindings between local ovs and onv-sb

Performance and Stability

This release is mainly about improving control plane performance, scalability, fast recovery and stability.

v0.10.0 -- 2019/12/23

Performance

• Update ovn to 2.12.0 and pick performance and raft bugfix from upstream
• Modify upstream ovn to reduce memory footprint
• CniServer filter pod in informer list-watch and disable resync
• Skip evicted pod when enqueueAddPod and enqueueUpdatePod
• When controller restart skip pod already create lsp
• As lr-route-add with --may-exist will replace exist route, no need for another delete

Monitoring

• Pinger support to check external address

Bugfix

• When all ip in subnet is used create lsp will panic with index out of range err
• Mount /var/run/netns into kube-ovn-cniserver for kind
• Use ep.subset.port.name to infer target port number
• Typo in start-ovs.sh
• When delete node recycle related ip/route resource
• Nbctl need timeout to avoid hang infinitely
• Block subnet deletion when there is any ip in use
• IP conflict when use ippool
• GC logical_switch_port form listing pods and nodes
• Do not add unallocated pod to port-group
• PodSelector in networkpolicy should only consider pods in the same ns

Mics

• Support kind installation
• Use label to select leader to avoid pod status misleading
• Add wait in cniserver and controller to reduce errors and restarts

v0.9.1 -- Bugfix

This release fix bugs found in v0.9.0

Bugfix

• When all ip in subnet is used create lsp will panic with index out of range err
• Mount /var/run/netns into kube-ovn-cniserver for kind
• Use ep.subset.port.name to infer target port number
• Typo in start-ovs.sh
• When delete node recycle related ip/route resource
• Nbctl need timeout to avoid hang infinitely
• Block subnet deletion when there any ip in use

v0.9.0 -- Visualization and Stability

This release is mainly about visualization, controller performance, stability and bugfix. Thanks for all the advice and bug report from the community

Monitoring

• Improve kube-ovn-pinger metrics to check apiserver and dns
• Add kube-ovn-controller metrics to show the controller status
• Add grafana templates to visualize metrics

Performance

• Adjust client-go param to increase parallelism
• Adjust ovn-db and ovn-controller resource
• Merge some ovn-nb requests and remove most wait=ovn-nb params

Stability and Bugfix

• LB init conflict when use multiple kube-ovn-controller
• Static Route might lost during leader election
• If pod have not a status.PodIP skip add/del static route
• Add keepalive to ovn-controller
• Add qlen when set egress QoS
• Add ingress_policing_burst to accurate limit ingress bandwidth
• GC resources when kube-ovn-controller starts
• Re-annotate related namespaces when subnet deleted.
• Check the short name of kubernetes services which is independent of the cluster domain name
• Daemonset updateStrategy changes to OnDelete for grace update
• Use new upstream ovn with some kube-ovn related modification

Misc

• Remove most privilege container
• When use nodelocaldns, do not nat the local dns server ip

v0.8.0

CHANGELOG

v0.8.0 -- 2019/10/08

Gateway

• Support active-backup mode centralized gateway high available

Diagnose Tools

• Kubectl plugin to trace/tcpdump/diagnose pod network traffic
• Pinger to test cluster network quality and expose metrics to Prometheus

IPAM

• Join subnet ip now can be displayed by kubectl get ip

Security

• Enable port security to prevent Mac and IP spoofing
• Allow nodes to pods traffic for private subnet

Mics

• Support hostport
• Update OVN/OVS to 2.11.3
• Update Go to 1.13

v0.7.0

IPAM

• Reserve vNic for statefulset pods, statefulset pod will reuse previous nic info during statefulset lifetime
• New IP CRD, now you can use kubectl get ip to obtain ip allocation info

Subnet

• Check logical switch existence before related operations
• Calculate default values for custom subnet
• Auto unbind previous subnet when namespace bind to a new subnet
• Subnet CRD now has status field to show ip allocation statistic and subnet condition
• Write subnet annotations back to bind namespace

Security

• Enable traffic mirror by default
• Support select all type NetworkPolicy rules
• Private subnet now applies acl to all ports not only gateway ports

IPv6

• Control plan components now can communicate with IPv6 protocol

Misc

• New logo
• 中文文档
• Test Kube-OVN compatible on CentOS 7.5/Ubuntu 16.04 and Ubuntu 18.04
• Add support for Kubespray and kubeasz installation tools
• Rename cni conf to 00-kube-ovn.conflist to improve kubelet priority
• Basic TCP performance test on aliyun.