How to use different LLM's and host everything locally.

[shawnZhang-3]

I tried to host everything locally followed the instructions:
https://github.com/arc53/DocsGPT/wiki/How-to-use-different-LLM's#hosting-everything-locally;but it doesn't work.

1.Firstly, I runned follow :

Use Manifest to host API locally: Just run in a seperate shell. You will need 80gb of extra space on your disk

python3 -m manifest.api.app
--model_type huggingface
--model_name_or_path bigscience/T0pp
--device 0

2. and get the url http://xxx.xxx.xxx.xxx:5000/

3.set the env file:
LLM_NAME=manifest
API_KEY=http://xxx.xxx.xxx.xxx:5000/
EMBEDDINGS_NAME=huggingface_sentence-transformers/all-mpnet-base-v2
EMBEDDINGS_KEY=http://xxx.xxx.xxx.xxx:5000/embed
VITE_API_STREAMING=false
CELERY_BROKER_URL=redis://localhost:6379/0
CELERY_RESULT_BACKEND=redis://localhost:6379/1
MONGO_URI=mongodb://localhost:27017/docsgpt
HUGGINGFACEHUB_API_TOKEN=http://xxx.xxx.xxx.xxx:5000

4.then I runned docker-compose build && docker-compose up, It worked well
5.when I uploaded file and pressed train

I got
"Did not find openai_api_key, please add an environment variable OPENAI_API_KEY which contains it, or pass openai_api_key as a named parameter. (type=value_error)", it seems like still used openai releated.

I don't know what's the problem and don't know how to do next step.

[bck01215]

I also have this issue. Were you able to find the solution?

[dartpain]

Also please note, default embeddings are locally run. so no need to EMBEDDINGS_KEY=http://xxx.xxx.xxx.xxx:5000/embed

Finally I recently added a swappable base_url for openai client, thus if you configure docsgpt with LLM_NAME=openai
You can run any model you want locally with openai compatible servers. for example vllm
Or ollama

But please make sure you put something in api_key like not-key
Also make sure you specify correct MODEL_NAME that you have launched via methods above or any other.
Also you will need to specify OPENAI_BASE_URL, if you are running it locally its probably http://localhost:11434/v1 for ollama. Also be careful of running docsgpt inside container as you will need to point OPENAI_BASE_URL to an address accessible by it

I will need to update our documentation to show a simple process

[gl2007]

Also please note, default embeddings are locally run. so no need to EMBEDDINGS_KEY=http://xxx.xxx.xxx.xxx:5000/embed

Finally I recently added a swappable base_url for openai client, thus if you configure docsgpt with LLM_NAME=openai You can run any model you want locally with openai compatible servers. for example vllm Or ollama

But please make sure you put something in api_key like not-key Also make sure you specify correct MODEL_NAME that you have launched via methods above or any other. Also you will need to specify OPENAI_BASE_URL, if you are running it locally its probably http://localhost:11434/v1 for ollama. Also be careful of running docsgpt inside container as you will need to point OPENAI_BASE_URL to an address accessible by it

I will need to update our documentation to show a simple process

A working example will be great