Monitoring metrics do not report external BGP peers

[salanki]

The Prometheus metric kube_router_controller_bgp_peers does not include eBGP peers. Some setups, ie. when peering with a ToR switch this is the only type of peer that is used. Monitoring proper function is impossible in these setups.

As the node can be in Ready state even with 0 BGP peers traffic from any pod scheduled on a node without the peer established will be blackholed, leading to service interruptions.

I'd suggest either including eBGP peers in the kube_router_controller_bgp_peers count, or adding a separate metric for it.

It would also make sense to add an option signal the CNI as NonReady until a peer has been established.

[murali-reddy]

@salanki thanks for suggestion. I am marking the bug for backlog of improvements.