How to deploy Llava in triton?

[forrestjgq]

Hello:

Glad to see that Llava is supported now. We're trying to deploy it in triton, how to do that?

[byshiue]

You could refer the document of Triton backend https://github.com/triton-inference-server/tensorrtllm_backend/blob/main/docs/llama.md and replace the engine and tokenizer by Llava if you already have a Llava engine.

[isaac-vidas]

Does tensorrt-llm-backend have support for multimodal? Is there an example for passing prompt and and an image through a request?

[byshiue]

There is no such example now.

[DefTruth]

same question. need some docs about how to deploy multimodal model (such as LLaVA) via triton server tensorrtllm_backend.

[Iven2132]

@DefTruth Did you figure it out? I'm looking for the same

[JeremySun1224]

There is no such example now.

multimodal