ReadinessCheck Support

[karthimohan]

Hi,

We have been using Marathon-consul for a while to render Haproxy config using Consul template. Recently we made change in our application to have endpoint for healthcheck to do a regular fast check < 1s to be used in Marathon. We have another endpoint which does a comprensive test which takes ~15S. Till this time, application is not ready to take the requests. So I read about Marathon readiness check and decided to go that route thinking that this will help us in terms of Consul registration based on the readiness check for the new instance. However with my testing it does not look it since my Marathon healthcheck passes <1s, it registers with Consul immediately and haproxy started getting the request.

Do we have a way to control this in Marathon-Consol to register the service only after the readinesscheck passes for the newly spin up instance?

[janisz]

Currently marathon-consul does not support readinnes checks.

Solution will be to copy readinnes check as a consul health check. Task will be registered when it's alive but marked as healthy when readinnes check passes. We need to tweak initial passing state when registering new service.

[karthimohan]

Thanks for the response @janisz As I mentioned above, we wanted to the readiness check endpoint to be checked only during the initial deployment and after that we have light weight healthcheck endpoint which will be checked at regular intervals by Marathon and Consul. If i add the readiness check endpoint as a Consul check, it will hit the endpoint which does comprehensive check every 10s. Sometimes it takes longer than the timeout interval 10s and task gets killed.

Below is the example
Consul HealthCheck
"SERVICE_9997_NAME": "service-dev-healthcheck",
"SERVICE_9997_CHECK_HTTP": "/healthcheck",
"SERVICE_9997_CHECK_INTERVAL": "10s",
"SERVICE_9997_CHECK_TIMEOUT": "10s",
Marathon Healthcheck
"healthChecks": [
{
"path": "/healthcheck",
"protocol": "HTTP",
"gracePeriodSeconds": 30,
"intervalSeconds": 10,
"portIndex": 2,
"timeoutSeconds": 10,
"maxConsecutiveFailures": 3
}
],
ReadinessChecks
"readinessChecks": [
{
"name": "DevReadyCheck",
"protocol": "HTTP",
"path": "/context/healthcheck",
"portName": "service-dev-http",
"intervalSeconds": 90,
"timeoutSeconds": 60,
"httpStatusCodesForReady": [ 200 ],
"preserveLastResponse": true
}
],

I may have misunderstood your suggestion. Please correct me if thats the case.

[janisz]

I don't know your code. How about lazy evaluation of readinnes endpoint. You can wrap endpoint with cache layer. If task was ready before then endpoint will immediately return 200 OK. In other case it will do regular expensive check and return its result.
If you can't cache readiness check response in application then problem is not trivial. Marathon does not expose event with readiness state (source). Then we need to parse Deployment event. We've been there and won't come back #129).