What --production flag does

[Lobosque]

And where is it implemented?

[parano]

Hi @Lobosque, great question! This should be explained better in the docs but I will try to answer it here.

When running bentoml serve without --production, it spins up a development server that focuses on dev experience. It runs all API code and runners within the same Python process which makes it easy to debug your code and find error messages. It also made it possible to use the --reload flag, which restarts the server if there's any code change or new models saved.

By adding --production, BentoML will treat it as a production deployment and will focus more on performance and scalability. In production mode, BentoML will manage Runners in a separate worker pool, configure replicas automatically based on hardware resources available(unless overridden by user configuration), and enable dynamic batching in front of Runner processes. The goal here is to automatically maximize resource utilization. This means that if you are running on a multi-core CPU or serving with GPU, BentoML makes sure that your serving workload is using all that compute resources efficiently. Note that BentoML also respects users' cgroup settings, so when deploying with Docker, you can limit how much resources each Bento server can use. This is what made large-scale deployment with Yatai on Kubernetes possible.

If you are interested in the implementation, check out the _internal module in BentoML source repo https://github.com/bentoml/BentoML/tree/main/bentoml/_internal, under the server, runner, and marshal directories.