Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement health checking to mark unhealthy backends with a zero weight #18

Open
SpComb opened this issue Aug 29, 2016 · 0 comments
Open

Comments

@SpComb
Copy link
Member

SpComb commented Aug 29, 2016

Currently clusterf relies on clusterf-docker to mark Docker containers as down and remove them from etcd as a crude form of backend health checking, which should be roughly sufficient to handle crashing server processes.

Using clusterf in any kind of serious production environment would require support for healthchecks. The ideal approach would be for each clusterf-ipvs node to perform its own healthchecks of its backends, and then mark unhealthy backends with an IPVS weight of zero. However, writing our healthchecks would be a bit of a pain, perhaps there is some existing golang package that we could use?

An alternative option would be the use of a more dedicated service discovery solution such as consul, with built-in health checks. However, the backend health seen by consul is not necessarily the same as seen by IPVS in the face of network partitions? Sadly, we can't really do any inline healthchecks on active backends connections as the kernel IPVS does not export any stats on failed connections in the form of e.g. TCP resets or timeouts.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant