Similarities/differences with Nvidia Triton Inference Server

[srinivasaraov]

I am exploring Promptflow and was thinking most of the functionality supported by Promotflow can be done using Ensembles or Business Logic Scripting in Nvidia Triton Inference Server. Can anyone throw some light on what are the pros and cons of implementing pipelines on Triton (BLS) vs Promptflow?

[0mza987]

@cloga Could you please help take a look at this question?

[cloga]

Hi @srinivasaraov ,
Prompt flow is E2E solution from Application developing, evaluation, deployment and post-deployment monitoring, it covered whole lifecycle.

But Nvidia Triton Inference Server is more focus on the Application or model hosting. Which is parity to managed online endpoint

Here are some advanced for managed online endpoint:

[dem108]

Hi, let me add some additional points.

Managed online endpoint on AzureML is the service to serve AI/ML models - which can use different inferencing stacks to serve them. That inferencing stack can be any - for example, AzureML's built-in inference server, Torchserve, TFServing, Triton Inference Server etc. Prompt flow deployment runs on top of built-in inference server.

Back to original question, prompt flow has story for end-to-end scenario as described by @cloga , deeply integrated with managed online endpoint and other parts of AzureML. Triton inference server also has good integration with managed online endpoint, with no-code deployment and bring your own container approach.

In addition, my understanding is that prompt flow targets to bring prompt engineering to extreme and support e2e (that goes beyond serving/inferencing), and that Triton inference server supports complex in-depth capabilities focusing on serving/inferencing.