Extremely slow agent with strange behaviour: After the tool finishes, the system inexplicably freezes for more than 30 seconds before it finally concludes the process. I shared my LangSmith session.

[mrctito]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

This is the Agent code:


async def chat_agent_ecommerce(query: str) -> str:
    try:
        agent_executor = await get_agent()

        resultado = await agent_executor.ainvoke(
                                {f"input": query, 
                                "dia_da_semana": datetime.now().strftime("%A"),
                                "data_hoje": datetime.now().strftime("%Y-%m-%d"),
                                "nome_cliente": "nome_usuario", 
                                "email_cliente": "email_usuario",
                                },
                                return_only_outputs=True)

        result_str = resultado["output"] if "output" in resultado else resultado
        return result_str
    except Exception as e:
        print(f"Erro: {str(e)}")


async def get_agent() -> str:
    memory = await GetMemory(input_key="input", output_key="output")
    prompt = get_prompt_agent_ecommerce_from_template()
    tools = get_chat_ecommerce_tools()
    tools_ex = [convert_to_openai_tool(t) for t in tools]
    tool_names = [tool.name for tool in tools]
    llm = cria_llm(app_keys)
    llm_with_tools = llm.bind_tools(tools_ex)
    llm_with_tools = llm_with_tools.bind(stop=["\nObservation", "\nFinal Answer"])

    agent = create_openai_tools_agent(llm=llm_with_tools,
                                      tools = tools, 
                                      prompt = prompt
                                      )
    
    agent_executor = AgentExecutor(agent=agent, 
                                    tools=tools,
                                    memory=memory, 
                                    max_iterations=15,
                                    max_execution_time=45,
                                    #early_stopping_method='generate',  # this does not work. it throws an exception
                                    early_stopping_method='force',
                                    verbose="True", 
                                    handle_parsing_errors=False,
                                    return_intermediate_steps=False
                                   )

    return agent_executor


def get_chat_ecommerce_tools():
    tools = []
    tools.append(pesquisar_produto)  #this is the main tool
    tools.append(acompanhar_pedido)
    tools.append(metodos_de_pagamento)
    tools.append(processar_pagamento_carrinho)
    tools.append(finalizar_compra_carrinho)
    tools.append(apagar_carrinho_compras)
    tools.append(criar_carrinho_compras)
    tools.append(adicionar_itens_carrinho_compras)
    tools.append(remover_item_carrinho_compras)
    tools.append(recuperar_carrinho_compras)

    print("Get app tools...")
    return tools


@tool
def pesquisar_produto(nome_produto:str) -> json:
    """
    útil para obter informações sobre produtos.

    Exemplos de perguntas que você pode responder com esta ferramenta:
     - "Você tem cadeiras de escritório?"
     - "Quero ver uma lista de todos os produtos que você tem."
     - "Quero ver cadeiras"
     - "Comprar mouse"
    """
    produtos = filtrar_produtos_detalhados(nome_produto=nome_produto, max_results=5)

    produtos_str = json.dumps(produtos, indent=4, ensure_ascii=False)
    return produtos_str

Description

I have an agent equipped with 10 tools. The primary tool is called "search_product," and it is used to search for products in a MySQL database. This tool is well-optimized and takes less than 3 seconds to read the database and load the information.

However, I'm encountering a peculiar issue that I've been unable to resolve: While the agent executes all steps swiftly, completing in less than 10 seconds, there is a significant delay when it returns the final message from the "search_product" tool. It pauses for up to 30 seconds before finishing.

To clarify further, the execution of the agent itself is fast and error-free, taking about 6 to 8 seconds in total. But after the tool finishes, although the console shows (because verbose=true) that the response is ready, the system inexplicably freezes for more than 30 seconds before it finally concludes the process.

Im using GPT-4-Turbo.

This is a LangSmith debug session:

I shared a LangsSmith session with a simple question: 'I want a new smartphone.' The agent runs only one tool, 'Search_Products,' and it takes more than 40 seconds:

LangSmith debug session

**I have been trying to resolve this for many weeks and I really need to improve this performance. I would truly appreciate any help!

I would also appreciate some assistance in migrating this code to LCEL.**

System Info

(.venv) mrctito@mrctito:~/desenv/prometeu/prometeu_service_server$ pip freeze | grep langchain
langchain==0.1.16
langchain-community==0.0.32
langchain-core==0.1.42
langchain-experimental==0.0.57
langchain-openai==0.1.3
langchain-text-splitters==0.0.1

(.venv) mrctito@mrctito:~/desenv/prometeu/prometeu_service_server$ python -m langchain_core.sys_info

System Information

OS: Linux
OS Version: #1 SMP Fri Mar 29 23:14:13 UTC 2024
Python Version: 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0]

Package Information

langchain_core: 0.1.42
langchain: 0.1.16
langchain_community: 0.0.32
langsmith: 0.1.31
langchain_experimental: 0.0.57
langchain_openai: 0.1.3
langchain_text_splitters: 0.0.1
langgraph: 0.0.36

Packages not installed (Not Necessarily a Problem)

The following packages were not found:

langserve

[dosubot[bot]]

To tackle the delay issue you're experiencing, start by enabling detailed logging for your agent's execution process. This can be done through the set_debug(True) or set_verbose(True) methods, which will help identify where the delay occurs by showing inputs and outputs at each step. For more comprehensive debugging, consider using tracing platforms like LangSmith for logging, visualizing, and debugging your LLM applications.

If detailed logging does not pinpoint the delay, examine the implementation of your agent's final steps or any custom callbacks that execute after the last tool finishes. There might be unoptimized callbacks or processing steps causing the delay, and optimizing these could resolve the issue.

Regarding migrating to LangChain Expression Language (LCEL), it's designed for seamless transition from prototypes to production, offering features like unified language model interaction, complex application building primitives, and both synchronous and asynchronous API call support. LCEL also enhances performance with batch operations, optimized parallel execution, and comprehensive logging and monitoring, which could significantly improve your application's efficiency.

For detailed guidance on LCEL and its benefits, refer to the LangChain documentation. This approach should help improve the performance of your agent and address the delay issue effectively.

Details

Sources

• Agent with retriever_tool is slow
• docs/docs/guides/development/debugging.md
• libs/langchain/langchain/agents/agent.py
• libs/langchain/langchain/agents/agent_iterator.py
• docs/docs/expression_language/index.mdx
• libs/langchain/langchain/runnables/init.py
• docs/docs/expression_language/why.ipynb
• libs/core/langchain_core/agents.py
• libs/experimental/langchain_experimental/llms/ollama_functions.py

About Dosu
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.

[mrctito]

@dosu It seems like you didn't read my message.

This is a LangSmith debug session:

I shared a LangsSmith session with a simple question: 'I want a new smartphone.' The agent runs only one tool, 'Search_Products,' and it takes more than 40 seconds:

LangSmith debug session

[mrctito]

@dosu Thank you for the suggestions. However, I'd like to clarify that the advice you provided is quite general and basic. I have already explored these avenues extensively. I'm reaching out for help because my knowledge of LangChain has reached its limits, and I've thoroughly researched the issues I'm encountering. If you have more advanced insights or specific guidance that goes beyond the standard procedures, I would greatly appreciate your input. This would be more beneficial given the complexity of the problems I'm currently facing with my implementation.

[eyurtsev]

@mrctito It looks like the latency is introduced by OpenAI based on this part of the trace.

AzureChatOpenAI is expected to be more reliable in terms of latency etc

[mrctito]

Yes, I agree that the latency seems to be introduced by OpenAI as indicated in the trace. However, I'm concerned whether LangChain might be making calls to OpenAI improperly, given that the response time is excessively long.

Could it be possible to verify if there's an issue with how LangChain is handling these API calls? I'm keen to understand if there's a more efficient way to manage these interactions to reduce the latency.

Thank you for any insights or suggestions you might have!

[Nerdthink]

Hi,
Have you been able to resolve this issue or have a walk around
@eyurtsev

[mrctito]

I am sorry. Unfortunately, I couldn’t solve the issue, so I decided to throw away my code and start again. The error is now gone.

[duob-ai]

I got the same problem when using an agent with tools.
Direct agent responses are fast as usual bot tool output is streamed extremely slow.

As in your case the latency seems to come from the llm. Can't figure out why.

Did you find a solution?

[Nerdthink]

Hi,
Have you been able to resolve this issue or have a walk around