TensorRT support

[bropines]

According to this branch from the regular SD automatic111 , it says that ControlNet is not supported. Is it possible to update the code with TensorRT support in advance, so that I don't have to wait for support afterwards?

[catboxanon]

From my understanding of how TensorRT support is currently, if CN were supported, the model would need to be compiled with the specific CN model(s) of choice, with a desired strength value, and input image, which then leaves you with a model strictly limited to those parameters. It's the same reason as if you wanted to use LoRA, the LoRA needs to be compiled or "baked" into the model instead, and the strength cannot be changed after the fact.

Obviously for something like CN this isn't viable since compiling a model takes, at minimum on good hardware, 15 minutes.

[bropines]

That is, all TensorRT models are specifically baked before use like some onnx, and you can't modify model weights using the CN method.

[MoreColors123]

From my understanding of how TensorRT support is currently, if CN were supported, the model would need to be compiled with the specific CN model(s) of choice, with a desired strength value, and input image, which then leaves you with a model strictly limited to those parameters. It's the same reason as if you wanted to use LoRA, the LoRA needs to be compiled or "baked" into the model instead, and the strength cannot be changed after the fact.

Obviously for something like CN this isn't viable since compiling a model takes, at minimum on good hardware, 15 minutes.

if this is the problem, then i would suggest a workflow where we prerender those models and fast-switch between them, depending on the use case. if it's faster, it's faster, and it would save so much time on inference. to render these controlnet models for some typical settings and switch between them with hotkeys or buttons in the ui would be a way to go.

I compiled different tensorRT models with different dimensions for a1111s tensorrt-extension already for example, because sometimes i want 6 pics with 448x448, sometimes i want 4 pics with 768x448. i' have about 80-100% performance gain by using those .trt.models, compared standard auto1111. many cool extensions worked from the start btw.: deforum, ultimate upscale, seed travel...

[vinch00]

Any plans here to support TensorRT for controlnet in A111?
If we can come up with a workflow to accomplish this even if it means baking in the controlnet with each model, could someone share how to do it with A111?