How to configure max_num_tokens and max_batch_size as runtime params?

[JoJoLev]

I noticed that these values can be configured at runtime, does this mean I must define them in the config.pbtxt if running on Triton?
Or should I pass the values as env variables of some sort since I am running this on Sagemaker?

Thanks!

[nv-guomingz]

Hi @Tabrizian would u please take a look this question?

[Tabrizian]

@JoJoLev For runtime values that are supported in model configuration you need to specify them in Triton's config.pbtxt file. There is documentation for the parameters that are supported here:

https://github.com/triton-inference-server/tensorrtllm_backend/blob/main/docs/model_config.md#common-inputs