Issue with Indexing in GraphRAG when Using Litellm as Proxy with Ollama

[octadion]

Hello,

I’m encountering an issue when using Litellm as a proxy in GraphRAG, specifically when routing requests to Ollama through Litellm during the indexing process.

Here’s the scenario:

• I’m using Litellm to route calls to Ollama for indexing in GraphRAG.
• When this setup is used, the output fails and shows logs like the following in Langfuse:

assistant: ".   .,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,"

• This causes the entire indexing process to fail.
• Interestingly, if I bypass Litellm and use Ollama directly with GraphRAG, the indexing works perfectly fine.
• Furthermore, if I use Litellm -> Ollama for query operations (asking questions), everything works without any problems.
  I need Litellm to handle token tracing via callback, which is why I’m trying to route the requests through it. Has anyone encountered this issue or have suggestions on why the indexing fails when routed through Litellm, but works fine for queries and direct usage? my model is llama 3 70 B btw

Any guidance or insights would be really helpful!

Thanks in advance!