Open WebUI LXC

[tteck]

If you selected to add "Ollama" during the Open WebUI installation

After setting up your user, all that's needed is to select a model. Click on the drop-down menu, then type llama3.2:1b in the "search for model" field (or choose from others available here).

To check if Ollama is running, enter your Open WebUI IP followed by :11434 in your browser.

[Deekerman]

I had to edit the .env for it to work

old

OLLAMA_BASE_URL=http://0.0.0.0:11434

new

OLLAMA_BASE_URL=http://<ip>:11434

then it worked

[Deekerman]

@havardthom Thank you. I have the drivers installed for the gpu, and nvidia-smi pulls up the gpu information. It just doesnt seem to be using it. So i wasnt sure if there was somewhere within open webui or ollama i needed to activate the use. or something specific i need from either of them for it to work.

For reference, im using a Quadro P2000
Driver Version: 550.67 CUDA Version: 12.4

[havardthom]

If run these in LXC console you should see some logs on ollama startup like which LLM runners it supports, GPU discovery and which device is used for inference.

systemctl restart ollama.service
journalctl -u ollama.service -n

Here's an example with Intel iGPU

level=INFO source=common.go:49 msg="Dynamic LLM libraries" runners="[cpu_avx cpu_avx2 oneapi cpu]"
018+01:00 level=INFO source=gpu.go:221 msg="looking for compatible GPUs"
122+01:00 level=INFO source=types.go:123 msg="inference compute" id=0 library=oneapi variant="" compute="" driver=0.0 name="Intel(R) UHD Graphics" total="0 B" available="0 B"

[Deekerman]

Well it does seem to find it, i dont know if its now or before but thats good its there.

Oct 29 15:31:34 openwebui ollama[2742]: time=2024-10-29T15:31:34.431-04:00 level=INFO source=common.go:49 msg="Dynamic LLM libraries" runners="[cuda_v12 rocm_v60102 cpu cpu_avx cpu_avx2 cuda_v11]"
Oct 29 15:31:34 openwebui ollama[2742]: time=2024-10-29T15:31:34.432-04:00 level=INFO source=gpu.go:221 msg="looking for compatible GPUs"
Oct 29 15:31:34 openwebui ollama[2742]: time=2024-10-29T15:31:34.637-04:00 level=INFO source=types.go:123 msg="inference compute" id=GPU-removed library=cuda variant=v12 compute=6..1 driver=12.4 name="Quadro P2000" total="4.9 GiB" available="4.7 GiB"

When i run a test, i guess i was expecting to see something pop up here, indicating it was used.
Im still pretty new to this so maybe im understanding it wrong.

Thank you for your help.

[mhoogenbosch]

Well it does seem to find it, i dont know if its now or before but thats good its there.

Oct 29 15:31:34 openwebui ollama[2742]: time=2024-10-29T15:31:34.431-04:00 level=INFO source=common.go:49 msg="Dynamic LLM libraries" runners="[cuda_v12 rocm_v60102 cpu cpu_avx cpu_avx2 cuda_v11]"
Oct 29 15:31:34 openwebui ollama[2742]: time=2024-10-29T15:31:34.432-04:00 level=INFO source=gpu.go:221 msg="looking for compatible GPUs"
Oct 29 15:31:34 openwebui ollama[2742]: time=2024-10-29T15:31:34.637-04:00 level=INFO source=types.go:123 msg="inference compute" id=GPU-removed library=cuda variant=v12 compute=6..1 driver=12.4 name="Quadro P2000" total="4.9 GiB" available="4.7 GiB"

When i run a test, i guess i was expecting to see something pop up here, indicating it was used. Im still pretty new to this so maybe im understanding it wrong.

Thank you for your help.

You need to run NVIDIA-SMI on the host. So in the proxmox ssh console, not on the LXC host. So maybe you're in luck and it just does work. Mine does, when checking the host, it shows the correct information. You need to 'ask' a question first.

[Deekerman]

@mhoogenbosch well it does show up there!

Thank you for the clarification and assistance. I thought it would show up in the lxc console but again, new to this. I got my emby working and tdarr and basically left it alone since then.

[tristan-k]

Please see my comment at the corresponding ollama issue.

[Popcorncandy09]

how can i setup SSO with authentik?