Does it work "only" with OpenAI's LLMs?

[vanetreg]

Does it work "only" with OpenAI LLMs or also with others like Sonnet 3.5, or any from Ollama?
If so, how? Pls. add info to docs / readme.

[codelion]

It only needs an OPENAI API compatible endpoint to work. Ollama already providers it, so you can use it with it directly (see here). For everything else you can use litellm proxy that supports all LLMs, it provides an OpenAI API compatible endpoint for Anthropic. So, you would just set that up and point optillm to that. I will add a note in the read about it.

[vanetreg]

The HF space is great, thank you, I'll use it for testing the techniques there.

Earlier you wrote we need not have to modify code, so I tried with dotenv using command:
python optillm.py --base_url https://openrouter.ai/api/v1 --api-key <your-openrouter-api-key> --model nousresearch/hermes-3-llama-3.1-405b:free
It starts, terminal shows:

2024-09-21 09:10:38,450 - INFO - Starting server with approach: auto
2024-09-21 09:10:38,451 - INFO - Server configuration: {'approach': 'auto', 'mcts_simulations': 2, 'mcts_exploration': 0.2, 'mcts_depth': 1, 'best_of_n': 3, 'model': 'nousresearch/hermes-3-llama-3.1-405b:free', 'rstar_max_depth': 3, 'rstar_num_rollouts': 5, 
'rstar_c': 1.4, 'n': 1, 'base_url': 'https://openrouter.ai/api/v1', 'api_key': '[REDACTED]', 'return_full_response': False, 'port': 8000, 'simulations': 2, 'exploration': 0.2, 'depth': 1}
 * Serving Flask app 'optillm'
 * Debug mode: off
2024-09-21 09:10:38,481 - INFO - WARNING: This is a development server. Do not use it 
in a production deployment. Use a production WSGI server instead.
 * Running on all addresses (0.0.0.0)
 * Running on http://127.0.0.1:8000
 * Running on http://192.168.0.15:8000
2024-09-21 09:10:38,482 - INFO - Press CTRL+C to quit
2024-09-21 09:11:23,429 - INFO - 127.0.0.1 - - [21/Sep/2024 09:11:23] "GET / HTTP/1.1" 401 -
2024-09-21 09:11:23,684 - INFO - 127.0.0.1 - - [21/Sep/2024 09:11:23] "GET /favicon.ico HTTP/1.1" 401 -

but checking the frontend at http://localhost:8000/ I got error:

{
  "error": "Invalid Authorization header. Expected format: 'Authorization: Bearer YOUR_API_KEY'"
}

[codelion]

As mentioned in the help the api-key is for authenticating the clients to the optillm proxy not the model provider. I can see how this can be confusing, I will add an api-key flag that can be used with base_url as you are doing and update this one to be called optillm-api-key to make it clear.

Meanwhile, you can just set the OPENAI_API_KEY environment variable with your <your-openrouter-api-key> before running the command and it will work.

[vanetreg]

After changing OPENAI_API_KEY environment variable with my "your-openrouter-api-key"
and using command:
python optillm.py --base_url https://openrouter.ai/api/v1 --model nousresearch/hermes-3-llama-3.1-405b:free
at http://localhost:8000/ I got 404:

Not Found
The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.

In terminal (in Cursor) it looks OK:

(optillm-venv2) PS D:\Projects\AI_testing\OptiLLM\optillm> python optillm.py --base_url https://openrouter.ai/api/v1  --model nousresearch/hermes-3-llama-3.1-405b:free
2024-09-21 10:53:55,873 - INFO - Starting server with approach: auto
2024-09-21 10:53:55,873 - INFO - Server configuration: {'approach': 'auto', 'mcts_simulations': 2, 'mcts_exploration': 0.2, 'mcts_depth': 1, 'best_of_n': 
3, 'model': 'nousresearch/hermes-3-llama-3.1-405b:free', 'rstar_max_depth': 3, 'rstar_num_rollouts': 5, 'rstar_c': 1.4, 'n': 1, 'base_url': 'https://openrouter.ai/api/v1', 'api_key': '', 'return_full_response': False, 'port': 8000, 'simulations': 2, 'exploration': 0.2, 'depth': 1}
 * Serving Flask app 'optillm'
 * Debug mode: off
2024-09-21 10:53:55,899 - INFO - WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
 * Running on all addresses (0.0.0.0)
 * Running on http://127.0.0.1:8000
 * Running on http://192.168.0.15:8000
2024-09-21 10:53:55,900 - INFO - Press CTRL+C to quit
2024-09-21 10:54:26,525 - INFO - 127.0.0.1 - - [21/Sep/2024 10:54:26] "GET / HTTP/1.1" 404 -
2024-09-21 10:54:28,534 - INFO - 127.0.0.1 - - [21/Sep/2024 10:54:28] "GET / HTTP/1.1" 404 -
2024-09-21 10:54:41,561 - INFO - 127.0.0.1 - - [21/Sep/2024 10:54:41] "GET / HTTP/1.1" 404 -
2024-09-21 10:56:06,580 - INFO - 192.168.0.15 - - [21/Sep/2024 10:56:06] "GET / HTTP/1.1" 404 -
2024-09-21 10:56:06,948 - INFO - 192.168.0.15 - - [21/Sep/2024 10:56:06] "GET /favicon.ico HTTP/1.1" 404 -
2024-09-21 10:56:15,358 - INFO - 127.0.0.1 - - [21/Sep/2024 10:56:15] "GET / HTTP/1.1" 404 -

[codelion]

The proxy doesn't have a web server. It only exposes a openai compatible api. So, you will need to call it using your own code or some front-end that can connect to an openai compatible api.

If you use this code you should be able to get the response.

There is a PR #19 under works to add a GUI for optillm. Once that is merged you will be able to browse to localhost and get a web ui to use with optillm.

[vanetreg]

Using your code slightly modified:

import os
from openai import OpenAI

OPENAI_KEY = os.environ.get("OPENAI_API_KEY")
OPENAI_BASE_URL = "http://localhost:8000/v1"

client = OpenAI(api_key=OPENAI_KEY, base_url=OPENAI_BASE_URL)

response = client.chat.completions.create(
  model="moa-meta-llama/llama-3.1-8b-instruct:free",
  messages=[
    {
      "role": "user",
      "content": "Write a Python program to print out the first 5 prime numbers"
    }
  ],
  temperature=0.2
)

print(response)

in backend terminal I got:

2024-09-21 11:36:31,326 - INFO - Received request to /v1/chat/completions
2024-09-21 11:36:31,802 - INFO - Using approach moa, with meta-llama/llama-3.1-8b-instruct:free
2024-09-21 11:36:32,401 - INFO - HTTP Request: POST https://openrouter.ai/api/v1/chat/completions "HTTP/1.1 200 OK"
2024-09-21 11:36:36,449 - ERROR - Error processing request: list index out of range
2024-09-21 11:36:36,452 - INFO - 127.0.0.1 - - [21/Sep/2024 11:36:36] "POST /v1/chat/completions HTTP/1.1" 500 -

Note: now in .env OPENAI_KEY = OPENROUTER_API_KEY
and backend started with:
python optillm.py --base_url https://openrouter.ai/api/v1/ --model meta-llama/llama-3.1-8b-instruct:free

[vanetreg]

I started proxy successfully with:
python optillm.py --base_url https://openrouter.ai/api/v1 --model nousresearch/hermes-3-llama-3.1-405b:free

having:
INFO - Server configuration: {'approach': 'auto', 'mcts_simulations': 2, 'mcts_exploration': 0.2, 'mcts_depth': 1, 'best_of_n': 3, 'model': 'nousresearch/hermes-3-llama-3.1-405b:free', 'rstar_max_depth': 3, 'rstar_num_rollouts': 5, 'rstar_c': 1.4, 'n': 1, 'base_url': 'https://openrouter.ai/api/v1', 'optillm_api_key': '', 'return_full_response': False, 'port': 8000, 'log': 'info', 'simulations': 2, 'exploration': 0.2, 'depth': 1}

and having client:

OPENAI_BASE_URL = "http://localhost:8000/v1"

# calling Openrouter's API
client = OpenAI(api_key=OPENROUTER_API_KEY, base_url=OPENAI_BASE_URL)

response = client.chat.completions.create(
  model="plansearch-nousresearch/hermes-3-llama-3.1-405b:free",
  messages=[
    {
      "role": "user",
      "content": "How many Rs are there in the word strawberry?"
    }
  ],
  temperature=0.2
)

but I still got 400 and 500, it calls the OpenAI API incorrectly instead of Openrouter's:

2024-10-05 10:01:13,426 - INFO - Received request to /v1/chat/completions
2024-10-05 10:01:13,864 - INFO - Using approach(es) ['plansearch'], operation SINGLE, with model nousresearch/hermes-3-llama-3.1-405b:free
2024-10-05 10:01:13,865 - INFO - Generating initial observations
2024-10-05 10:01:14,090 - INFO - HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 400 Bad Request"
2024-10-05 10:01:14,092 - ERROR - Error processing request: Error code: 400 - {'error': {'message': 'invalid model ID', 'type': 'invalid_request_error', 'param': None, 'code': None}}
2024-10-05 10:01:14,094 - INFO - 127.0.0.1 - - [05/Oct/2024 10:01:14] "POST /v1/chat/completions HTTP/1.1" 500 -

Any help is appreciated :)

[codelion]

2024-10-05 11:07:59,476 - INFO - HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 400 Bad Request"

suggests that the api is going to the OpenAI endpoint not the base_url you have set. I just pushed a change 575692f check if you have that in your code.

Otherwise, can you try setting the export OPENAI_API_KEY=<youropenrouterkey> before running the proxy:

python optillm.py --base_url https://openrouter.ai/api/v1

Then call it from the client and try. This will ensure that the proxy reads the environment variable from the env instead of the client.
Also check if you have any .env files as that would override the settings you do in the environment variable.

[vanetreg]

I was incorrectly updating your code :(
I completed it now and optillm looks working with Openrouter, even though I have:

  File "D:\***v\Lib\site-packages\openai\_base_client.py", line 1051, in _request
    raise self._make_status_error_from_response(err.response) from None
openai.APIStatusError: Error code: 402 - {'error': {'message': 'You are either out of credits, or requesting a resource that requires owning at least a few. Please visit https://openrouter.ai/credits to add more.', 'code': 402}, 'user_id': 'user_***'}

which
'You are either out of credits, or requesting a resource that requires owning at least a few."
is weird, because I used yesterday the same free LLM with another project without any such credit warnings and issues.

[codelion]

You can check the activity here https://openrouter.ai/activity it should show the models you were trying out.

[vanetreg]

I guess I'm above the daily limit, yesterday testing Pythagora with this LLM.
"Free limit: If you are using a free model variant (with an ID ending in :free), then you will be limited to 20 requests per minute and 200 requests per day"
Pls. consider my idea having a forkable studio template at LightningAI
https://lightning.ai/studios
which uses OptiLLM with Ollama!.

Thank you for your help ! :)

[codelion]

That's a good idea tracking it here - #56