Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

raise asyncio.TimeoutError from None TimeoutError #654

Open
paulge1021 opened this issue Sep 6, 2024 · 2 comments
Open

raise asyncio.TimeoutError from None TimeoutError #654

paulge1021 opened this issue Sep 6, 2024 · 2 comments
Labels
bug Something isn't working module/ai-service ai-service related

Comments

@paulge1021
Copy link

Describe the bug
In Home tab at Ask input Question, return "Internal server error"

To Reproduce
Steps to reproduce the behavior:

  1. Go to 'Home' tab
  2. Input Question
  3. Click on Ask
  4. See error 'Internal server error'

Expected behavior
raise asyncio.TimeoutError from None
TimeoutError

May be open the Timeout Seting

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • OS: [windows 10,docke rdesktop]
  • Browser [chrome]

Wren AI Information

  • Version: [0.9.2]
  • LLM_PROVIDER= ollama_llm
  • GENERATION_MODEL= llama3.1:70b

Additional context
wrenai-wren-ai-service.log

generate [src.pipelines.ask_details.generation.generate()] encountered an error<
Node inputs:
{'generator': '<src.providers.llm.ollama.AsyncGenerator object at...',
'prompt': "<Task finished name='Task-1067' coro=<AsyncGraphAd..."}

Traceback (most recent call last):
File "/app/.venv/lib/python3.12/site-packages/hamilton/async_driver.py", line 122, in new_fn
await fn(**fn_kwargs) if asyncio.iscoroutinefunction(fn) else fn(**fn_kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/src/utils.py", line 118, in wrapper_timer
return await process(func, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/src/utils.py", line 102, in process
return await func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/.venv/lib/python3.12/site-packages/langfuse/decorators/langfuse_decorator.py", line 182, in async_wrapper
self._handle_exception(observation, e)
File "/app/.venv/lib/python3.12/site-packages/langfuse/decorators/langfuse_decorator.py", line 422, in _handle_exception
raise e
File "/app/.venv/lib/python3.12/site-packages/langfuse/decorators/langfuse_decorator.py", line 180, in async_wrapper
result = await func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/src/pipelines/ask_details/generation.py", line 61, in generate
return await generator.run(prompt=prompt.get("prompt"))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/src/providers/llm/ollama.py", line 109, in run
response = await session.post(
^^^^^^^^^^^^^^^^^^^
File "/app/.venv/lib/python3.12/site-packages/aiohttp/client.py", line 608, in _request
await resp.start(conn)
File "/app/.venv/lib/python3.12/site-packages/aiohttp/client_reqrep.py", line 971, in start
with self._timer:
File "/app/.venv/lib/python3.12/site-packages/aiohttp/helpers.py", line 735, in exit
raise asyncio.TimeoutError from None
TimeoutError

Oh no an error! Need help with Hamilton?
Join our slack and ask for help! https://join.slack.com/t/hamilton-opensource/shared_invite/zt-1bjs72asx-wcUTgH7q7QX1igiQ5bbdcg

2024-09-05 05:41:11,206 - wren-ai-service - ERROR - ask-details pipeline - OTHERS: (ask_details.py:112)
Traceback (most recent call last):
File "/src/web/v1/services/ask_details.py", line 90, in ask_details
generation_result = await self._pipelines["generation"].run(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/src/utils.py", line 118, in wrapper_timer
return await process(func, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/src/utils.py", line 102, in process
return await func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/.venv/lib/python3.12/site-packages/langfuse/decorators/langfuse_decorator.py", line 182, in async_wrapper
self._handle_exception(observation, e)
File "/app/.venv/lib/python3.12/site-packages/langfuse/decorators/langfuse_decorator.py", line 422, in _handle_exception
raise e
File "/app/.venv/lib/python3.12/site-packages/langfuse/decorators/langfuse_decorator.py", line 180, in async_wrapper
result = await func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/src/pipelines/ask_details/generation.py", line 114, in run
return await self._pipe.execute(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/.venv/lib/python3.12/site-packages/hamilton/async_driver.py", line 368, in execute
raise e
File "/app/.venv/lib/python3.12/site-packages/hamilton/async_driver.py", line 359, in execute
outputs = await self.raw_execute(final_vars, overrides, display_graph, inputs=inputs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/.venv/lib/python3.12/site-packages/hamilton/async_driver.py", line 320, in raw_execute
raise e
File "/app/.venv/lib/python3.12/site-packages/hamilton/async_driver.py", line 315, in raw_execute
results = await await_dict_of_tasks(task_dict)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/.venv/lib/python3.12/site-packages/hamilton/async_driver.py", line 23, in await_dict_of_tasks
coroutines_gathered = await asyncio.gather(*coroutines)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/.venv/lib/python3.12/site-packages/hamilton/async_driver.py", line 36, in process_value
return await val
^^^^^^^^^
File "/app/.venv/lib/python3.12/site-packages/hamilton/async_driver.py", line 91, in new_fn
fn_kwargs = await await_dict_of_tasks(task_dict)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/.venv/lib/python3.12/site-packages/hamilton/async_driver.py", line 23, in await_dict_of_tasks
coroutines_gathered = await asyncio.gather(*coroutines)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/.venv/lib/python3.12/site-packages/hamilton/async_driver.py", line 36, in process_value
return await val
^^^^^^^^^
File "/app/.venv/lib/python3.12/site-packages/hamilton/async_driver.py", line 122, in new_fn
await fn(**fn_kwargs) if asyncio.iscoroutinefunction(fn) else fn(**fn_kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/src/utils.py", line 118, in wrapper_timer
return await process(func, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/src/utils.py", line 102, in process
return await func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/.venv/lib/python3.12/site-packages/langfuse/decorators/langfuse_decorator.py", line 182, in async_wrapper
self._handle_exception(observation, e)
File "/app/.venv/lib/python3.12/site-packages/langfuse/decorators/langfuse_decorator.py", line 422, in _handle_exception
raise e
File "/app/.venv/lib/python3.12/site-packages/langfuse/decorators/langfuse_decorator.py", line 180, in async_wrapper
result = await func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/src/pipelines/ask_details/generation.py", line 61, in generate
return await generator.run(prompt=prompt.get("prompt"))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/src/providers/llm/ollama.py", line 109, in run
response = await session.post(
^^^^^^^^^^^^^^^^^^^
File "/app/.venv/lib/python3.12/site-packages/aiohttp/client.py", line 608, in _request
await resp.start(conn)
File "/app/.venv/lib/python3.12/site-packages/aiohttp/client_reqrep.py", line 971, in start
with self._timer:
File "/app/.venv/lib/python3.12/site-packages/aiohttp/helpers.py", line 735, in exit
raise asyncio.TimeoutError from None
TimeoutError
INFO: 172.18.0.6:37524 - "GET /v1/ask-details/cbffda05-060c-4d60-955f-efae1116d820/result HTTP/1.1" 200 OK
@paulge1021 paulge1021 added the bug Something isn't working label Sep 6, 2024
@cyyeh
Copy link
Member

cyyeh commented Sep 10, 2024

@paulge1021 thanks for reporting this issue. The error is due to llm model timeout. We'll fix this issue and expose environment variables(LLM_TIMEOUT, EMBEDDER_TIMEOUT) for users to fill in custom timeout values

@cyyeh cyyeh added the module/ai-service ai-service related label Sep 10, 2024
@camilleconte8
Copy link

I am struggling with this too. I have set the LLM and embedder timeouts to 600 in all of the .env files as well as in my ollama.py file, and I get this log of errors in my wren-ai-service container:

Click to view Error Log 
2024-09-19 14:14:18 ********************************************************************************
2024-09-19 14:14:18 > filter_columns_in_tables [src.pipelines.ask.retrieval.filter_columns_in_tables()] encountered an error<
2024-09-19 14:14:18 > Node inputs:
2024-09-19 14:14:18 {'prompt': "<Task finished name='Task-93' coro=<AsyncGraphAdap...",
2024-09-19 14:14:18  'table_columns_selection_generator': '<src.providers.llm.ollama.AsyncGenerator '
2024-09-19 14:14:18                                       'object at...'}
2024-09-19 14:14:18 ********************************************************************************
2024-09-19 14:14:18 Traceback (most recent call last):
2024-09-19 14:14:18   File "/app/.venv/lib/python3.12/site-packages/hamilton/async_driver.py", line 122, in new_fn
2024-09-19 14:14:18     await fn(**fn_kwargs) if asyncio.iscoroutinefunction(fn) else fn(**fn_kwargs)
2024-09-19 14:14:18     ^^^^^^^^^^^^^^^^^^^^^
2024-09-19 14:14:18   File "/src/utils.py", line 119, in wrapper_timer
2024-09-19 14:14:18     return await process(func, *args, **kwargs)
2024-09-19 14:14:18            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-19 14:14:18   File "/src/utils.py", line 103, in process
2024-09-19 14:14:18     return await func(*args, **kwargs)
2024-09-19 14:14:18            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-19 14:14:18   File "/app/.venv/lib/python3.12/site-packages/langfuse/decorators/langfuse_decorator.py", line 188, in async_wrapper
2024-09-19 14:14:18     self._handle_exception(observation, e)
2024-09-19 14:14:18   File "/app/.venv/lib/python3.12/site-packages/langfuse/decorators/langfuse_decorator.py", line 428, in _handle_exception
2024-09-19 14:14:18     raise e
2024-09-19 14:14:18   File "/app/.venv/lib/python3.12/site-packages/langfuse/decorators/langfuse_decorator.py", line 186, in async_wrapper
2024-09-19 14:14:18     result = await func(*args, **kwargs)
2024-09-19 14:14:18              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-19 14:14:18   File "/src/pipelines/ask/retrieval.py", line 233, in filter_columns_in_tables
2024-09-19 14:14:18     return await table_columns_selection_generator.run(prompt=prompt.get("prompt"))
2024-09-19 14:14:18            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-19 14:14:18   File "/src/providers/llm/ollama.py", line 109, in run
2024-09-19 14:14:18     response = await session.post(
2024-09-19 14:14:18                ^^^^^^^^^^^^^^^^^^^
2024-09-19 14:14:18   File "/app/.venv/lib/python3.12/site-packages/aiohttp/client.py", line 684, in _request
2024-09-19 14:14:18     await resp.start(conn)
2024-09-19 14:14:18   File "/app/.venv/lib/python3.12/site-packages/aiohttp/client_reqrep.py", line 994, in start
2024-09-19 14:14:18     with self._timer:
2024-09-19 14:14:18   File "/app/.venv/lib/python3.12/site-packages/aiohttp/helpers.py", line 713, in __exit__
2024-09-19 14:14:18     raise asyncio.TimeoutError from None
2024-09-19 14:14:18 TimeoutError
2024-09-19 14:14:18 -------------------------------------------------------------------
2024-09-19 14:14:18 Oh no an error! Need help with Hamilton?
2024-09-19 14:14:18 Join our slack and ask for help! https://join.slack.com/t/hamilton-opensource/shared_invite/zt-1bjs72asx-wcUTgH7q7QX1igiQ5bbdcg
2024-09-19 14:14:18 -------------------------------------------------------------------
2024-09-19 14:14:18 
2024-09-19 14:14:18 2024-09-19 18:14:18,583 - wren-ai-service - ERROR - ask pipeline - OTHERS:  (ask.py:307)
2024-09-19 14:14:18 Traceback (most recent call last):
2024-09-19 14:14:18   File "/src/web/v1/services/ask.py", line 145, in ask
2024-09-19 14:14:18     retrieval_result = await self._pipelines["retrieval"].run(
2024-09-19 14:14:18                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-19 14:14:18   File "/src/utils.py", line 119, in wrapper_timer
2024-09-19 14:14:18     return await process(func, *args, **kwargs)
2024-09-19 14:14:18            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-19 14:14:18   File "/src/utils.py", line 103, in process
2024-09-19 14:14:18     return await func(*args, **kwargs)
2024-09-19 14:14:18            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-19 14:14:18   File "/app/.venv/lib/python3.12/site-packages/langfuse/decorators/langfuse_decorator.py", line 188, in async_wrapper
2024-09-19 14:14:18     self._handle_exception(observation, e)
2024-09-19 14:14:18   File "/app/.venv/lib/python3.12/site-packages/langfuse/decorators/langfuse_decorator.py", line 428, in _handle_exception
2024-09-19 14:14:18     raise e
2024-09-19 14:14:18   File "/app/.venv/lib/python3.12/site-packages/langfuse/decorators/langfuse_decorator.py", line 186, in async_wrapper
2024-09-19 14:14:18     result = await func(*args, **kwargs)
2024-09-19 14:14:18              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-19 14:14:18   File "/src/pipelines/ask/retrieval.py", line 341, in run
2024-09-19 14:14:18     return await self._pipe.execute(
2024-09-19 14:14:18            ^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-19 14:14:18   File "/app/.venv/lib/python3.12/site-packages/hamilton/async_driver.py", line 368, in execute
2024-09-19 14:14:18     raise e
2024-09-19 14:14:18   File "/app/.venv/lib/python3.12/site-packages/hamilton/async_driver.py", line 359, in execute
2024-09-19 14:14:18     outputs = await self.raw_execute(final_vars, overrides, display_graph, inputs=inputs)
2024-09-19 14:14:18               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-19 14:14:18   File "/app/.venv/lib/python3.12/site-packages/hamilton/async_driver.py", line 320, in raw_execute
2024-09-19 14:14:18     raise e
2024-09-19 14:14:18   File "/app/.venv/lib/python3.12/site-packages/hamilton/async_driver.py", line 315, in raw_execute
2024-09-19 14:14:18     results = await await_dict_of_tasks(task_dict)
2024-09-19 14:14:18               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-19 14:14:18   File "/app/.venv/lib/python3.12/site-packages/hamilton/async_driver.py", line 23, in await_dict_of_tasks
2024-09-19 14:14:18     coroutines_gathered = await asyncio.gather(*coroutines)
2024-09-19 14:14:18                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-19 14:14:18   File "/app/.venv/lib/python3.12/site-packages/hamilton/async_driver.py", line 36, in process_value
2024-09-19 14:14:18     return await val
2024-09-19 14:14:18            ^^^^^^^^^
2024-09-19 14:14:18   File "/app/.venv/lib/python3.12/site-packages/hamilton/async_driver.py", line 91, in new_fn
2024-09-19 14:14:18     fn_kwargs = await await_dict_of_tasks(task_dict)
2024-09-19 14:14:18                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-19 14:14:18   File "/app/.venv/lib/python3.12/site-packages/hamilton/async_driver.py", line 23, in await_dict_of_tasks
2024-09-19 14:14:18     coroutines_gathered = await asyncio.gather(*coroutines)
2024-09-19 14:14:18                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-19 14:14:18   File "/app/.venv/lib/python3.12/site-packages/hamilton/async_driver.py", line 36, in process_value
2024-09-19 14:14:18     return await val
2024-09-19 14:14:18            ^^^^^^^^^
2024-09-19 14:14:18   File "/app/.venv/lib/python3.12/site-packages/hamilton/async_driver.py", line 122, in new_fn
2024-09-19 14:14:18     await fn(**fn_kwargs) if asyncio.iscoroutinefunction(fn) else fn(**fn_kwargs)
2024-09-19 14:14:18     ^^^^^^^^^^^^^^^^^^^^^
2024-09-19 14:14:18   File "/src/utils.py", line 119, in wrapper_timer
2024-09-19 14:14:18     return await process(func, *args, **kwargs)
2024-09-19 14:14:18            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-19 14:14:18   File "/src/utils.py", line 103, in process
2024-09-19 14:14:18     return await func(*args, **kwargs)
2024-09-19 14:14:18            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-19 14:14:18   File "/app/.venv/lib/python3.12/site-packages/langfuse/decorators/langfuse_decorator.py", line 188, in async_wrapper
2024-09-19 14:14:18     self._handle_exception(observation, e)
2024-09-19 14:14:18   File "/app/.venv/lib/python3.12/site-packages/langfuse/decorators/langfuse_decorator.py", line 428, in _handle_exception
2024-09-19 14:14:18     raise e
2024-09-19 14:14:18   File "/app/.venv/lib/python3.12/site-packages/langfuse/decorators/langfuse_decorator.py", line 186, in async_wrapper
2024-09-19 14:14:18     result = await func(*args, **kwargs)
2024-09-19 14:14:18              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-19 14:14:18   File "/src/pipelines/ask/retrieval.py", line 233, in filter_columns_in_tables
2024-09-19 14:14:18     return await table_columns_selection_generator.run(prompt=prompt.get("prompt"))
2024-09-19 14:14:18            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-19 14:14:18   File "/src/providers/llm/ollama.py", line 109, in run
2024-09-19 14:14:18     response = await session.post(
2024-09-19 14:14:18                ^^^^^^^^^^^^^^^^^^^
2024-09-19 14:14:18   File "/app/.venv/lib/python3.12/site-packages/aiohttp/client.py", line 684, in _request
2024-09-19 14:14:18     await resp.start(conn)
2024-09-19 14:14:18   File "/app/.venv/lib/python3.12/site-packages/aiohttp/client_reqrep.py", line 994, in start
2024-09-19 14:14:18     with self._timer:
2024-09-19 14:14:18   File "/app/.venv/lib/python3.12/site-packages/aiohttp/helpers.py", line 713, in __exit__
2024-09-19 14:14:18     raise asyncio.TimeoutError from None
2024-09-19 14:14:18 TimeoutError

Any suggestions?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working module/ai-service ai-service related
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@cyyeh @paulge1021 @camilleconte8