How to know when a gunicorn worker's warm up ends

[chlee1016]

Hi team!

I am writing to ask a question below.
Main question. Is there any way to know when a gunicorn worker warm up ends?

In our case, we write a file in a model's init function to see if the model containerized by BentoML is ready to receive requests.
However, after the model finish to write a file to check its readiness, it is still warming up, so there is a problem with high CPU usage which leads to Read timed out error.

Q. Is it right that the increase in CPU usage is due to the warm-up of the gunicorn worker?
Q. If so, is there any way to know when a gunicorn worker warm up ends?
Q. Or any ideas to solve this problem?

Many thanks!

[parano]

Q. Is it right that the increase in CPU usage is due to the warm-up of the gunicorn worker?
Q. If so, is there any way to know when a gunicorn worker warm up ends?
Q. Or any ideas to solve this problem?

Could you share a bit more about what your service init function does? It's hard to say if the increase in CPU usage is related to worker warm-up but might have to do with user code. I'd suggest adding some logging/tracing code in the service init function to understand the service warm-up process.

[chlee1016]

• Problem : when pod starts to be ready state on k8s, high CPU usage has been observed which leads to Read timed out error.

  • Initially I thought this was a gunicorn worker's warm up (not by user code)
  • but later realized that there is a mistake for the readiness check. It has described below.

• Description of checking readiness of pod

%%writefile predictor.py
class Predictor(object):
    def __init__(self):
        """
        Load your model files
        """
        pass
    def predict(self, df):
        return 1

%%writefile my_classifier.py
import pandas as pd

from bentoml import env, api, BentoService
from bentoml.adapters import DataframeInput
from predictor import Predictor

@env(infer_pip_packages=True)
class MyClassifier(BentoService):
    """
    A minimum prediction service
    """
    def __init__(self)):
        self.predictor=Predictor()
        # The code below is for checking readiness of k8s pod
        with open("/tmp/ready.check", "w") as f:
            f.write("This model is ready for serving")

    @api(input=DataframeInput(), batch=True)
    def predict(self, df: pd.DataFrame):
        """
        My inference API
        """
        return self.predictor.predict(df)

After save() our bento service, we can start our inference API server (which is on k8s).

docker build -t my-model <MY_BENTO_SERVICE_BUNDLE_NAME>
docker run -p 5000:5000 my-model --workers=4

If we set --workers=4, the logs of pod are as follows when launching.

[2021-08-27 14:49:24,210] INFO:bentoml.server:Starting BentoML proxy in production mode..
[2021-08-27 14:49:24,210] INFO:bentoml.server:Starting BentoML API server in production mode..
[2021-08-27 14:49:24,419] INFO:bentoml.marshal:Running micro batch service on :8080
[2021-08-27 14:49:24 +0900] [56] [INFO] Starting gunicorn 20.1.0
[2021-08-27 14:49:24 +0900] [56] [INFO] Listening at: http://0.0.0.0:56857 (56)
[2021-08-27 14:49:24 +0900] [56] [INFO] Using worker: sync
[2021-08-27 14:49:24 +0900] [57] [INFO] Booting worker with pid: 57
[2021-08-27 14:49:24 +0900] [1] [INFO] Starting gunicorn 20.1.0
[2021-08-27 14:49:24 +0900] [1] [INFO] Listening at: http://0.0.0.0:8080 (1)
[2021-08-27 14:49:24 +0900] [1] [INFO] Using worker: aiohttp.worker.GunicornWebWorker
[2021-08-27 14:49:24 +0900] [58] [INFO] Booting worker with pid: 58
[2021-08-27 14:49:24 +0900] [59] [INFO] Booting worker with pid: 59
[2021-08-27 14:49:24 +0900] [60] [INFO] Booting worker with pid: 60
[2021-08-27 14:49:24 +0900] [61] [INFO] Booting worker with pid: 61

The point is that we are determining the readiness of pod from when /tmp/ready.check is created. However, this means that the first worker out of 4 is ready, not all workers are ready(the other workers are still loading their model files as the code above). So now comes my question.

• Q. Could you tell me where the proper location to write /tmp/ready.check file is?
  • not sure, guessing here
  • I want to write the file when the execution of all workers' init function is finished.
• BentoML version : v0.13.1

[chlee1016]

I have solved this problem by customizing k8s readiness check logic and the steps for solution are as below.

Problem : when pod starts to be ready state on k8s, high CPU usage has been observed which leads to Read timed out error.

1. Each gunicorn worker write /tmp/ready_<pid>.check file when they launching.
2. The readiness probe of k8s pod checks if the number of files is same as the number of gunicorn workers.
3. Set enough time before readiness probe using readiness_probe_delay.
4. The pod can be changed smoothly and stably after warm up time.