[bug]upload file error #11710

chenzikun · 2024-12-16T13:39:34Z

Self Checks

This is only for bug report, if you would like to ask a question, please head to Discussions.
I have searched for existing issues search for existing issues, including closed ones.
I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
[FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
Please do not modify this template :) and fill in all the required fields.

Dify version

0.14.0

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

.env

storage_type=s3

opendal.UnexpectedError: Unexpected (temporary) at Writer::close, context: { url: https://s3.cn-northwest-1.amazonaws.com/llm-dify/upload_files/c463a29f-f080-42ab-adda-ec632932d9bc/28eb80f4-4c7f-4a1d-8691-81eafc94856c.png, called: http_util::Client::send, service: s3, path: upload_files/c463a29f-f080-42ab-adda-ec632932d9bc/28eb80f4-4c7f-4a1d-8691-81eafc94856c.png, written: 535646 } => send http request, source: error sending request for url (https://s3.cn-northwest-1.amazonaws.com/llm-dify/upload_files/c463a29f-f080-42ab-adda-ec632932d9bc/28eb80f4-4c7f-4a1d-8691-81eafc94856c.png)

✔️ Expected Behavior

No response

❌ Actual Behavior

No response

The text was updated successfully, but these errors were encountered:

chenzikun · 2024-12-16T13:42:26Z

add context

opendal.UnexpectedError: Unexpected (temporary) at Writer::close, context: { url: https://s3.cn-northwest-1.amazonaws.com/llm-dify/upload_files/334df837-1844-4405-a44e-64c528e819b6/78c43d9b-9edb-4679-b882-c7f776fd382f.png, called: http_util::Client::send, service: s3, path: upload_files/334df837-1844-4405-a44e-64c528e819b6/78c43d9b-9edb-4679-b882-c7f776fd382f.png, written: 318298 } => send http request, source: error sending request for url (https://s3.cn-northwest-1.amazonaws.com/llm-dify/upload_files/334df837-1844-4405-a44e-64c528e819b6/78c43d9b-9edb-4679-b882-c7f776fd382f.png)
2024-12-16 13:37:00,241.241 INFO [Dummy-34] [_client.py:1038] - HTTP Request: POST http://ai-llms-proxy-enetestus.autel.com/v1/chat/completions "HTTP/1.1 200 OK"
2024-12-16 13:37:11,390.390 ERROR [Dummy-35] [ext_storage.py:84] - Failed to save file upload_files/c463a29f-f080-42ab-adda-ec632932d9bc/28eb80f4-4c7f-4a1d-8691-81eafc94856c.png
Traceback (most recent call last):
  File "/app/api/extensions/ext_storage.py", line 82, in save
    self.storage_runner.save(filename, data)
  File "/app/api/extensions/storage/opendal_storage.py", line 37, in save
    self.op.write(path=filename, bs=data)

dosubot · 2024-12-16T13:47:44Z

Hey @chenzikun! I'm here to help you with bugs, questions, and becoming a contributor. Let's get to the bottom of this together!

I found a similar unsolved discussion regarding file URL mismatches in a self-hosted Dify instance using S3, which might be related to your issue. You can view the discussion here ^[1].

Regarding the error you're encountering, the opendal.UnexpectedError typically indicates a temporary issue when closing the writer during a file upload to S3. This could be due to network instability, temporary S3 service issues, or incorrect configuration settings.

Here are a few steps you can take to troubleshoot and potentially resolve the issue:

Check Network Stability: Ensure that your network connection is stable and there are no interruptions during the file upload process.
Verify S3 Configuration: Double-check your S3 configuration in the .env file to ensure all details like S3_ENDPOINT, S3_BUCKET_NAME, S3_ACCESS_KEY, S3_SECRET_KEY, and S3_REGION are correct and match your S3 setup ^[2].
Retry Mechanism: If the error is indeed temporary, implementing a retry mechanism in your code might help overcome transient network or service issues.
Logs and Monitoring: Enable detailed logging to capture more information about the error. This can help identify if there are specific patterns or conditions under which the error occurs.

If the issue persists, further investigation into the specific error messages and logs might be necessary to pinpoint the exact cause.

To continue talking to Dosu, mention @dosu.

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

chenzikun · 2024-12-16T13:50:07Z

api/extensions/ext_storage.py

 def get_storage_factory(storage_type: str) -> Callable[[], BaseStorage]:
     match storage_type:
         case StorageType.S3:
             from extensions.storage.opendal_storage import OpenDALStorage

             kwargs = _load_s3_storage_kwargs()
             return lambda: OpenDALStorage(scheme=OpenDALScheme.S3, **kwargs)
         case StorageType.OPENDAL:
             from extensions.storage.opendal_storage import OpenDALStorage

             scheme = OpenDALScheme(dify_config.STORAGE_OPENDAL_SCHEME)
             kwargs = _load_opendal_storage_kwargs(scheme)
             return lambda: OpenDALStorage(scheme=scheme, **kwargs)
         case StorageType.LOCAL:
             from extensions.storage.opendal_storage import OpenDALStorage

why case StorageType.S3, still use OpenDALStorage?

dosubot · 2024-12-16T13:51:30Z

The get_storage_factory function uses OpenDALStorage for case StorageType.S3 because it is designed to handle S3 storage using the OpenDAL framework. This setup allows OpenDALStorage to manage S3 storage operations by configuring it with the necessary S3 parameters such as bucket, endpoint, access_key_id, secret_access_key, and region, which are loaded using the _load_s3_storage_kwargs function. This approach ensures that OpenDALStorage can be used flexibly with different S3-compatible services, including handling special cases like AWS managed IAM and Cloudflare R2 ^[1].

To continue talking to Dosu, mention @dosu.

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

chenzikun · 2024-12-16T14:22:30Z

second bug cause by the change code

Traceback (most recent call last):
  File "/app/api/.venv/lib/python3.12/site-packages/flask/app.py", line 917, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/.venv/lib/python3.12/site-packages/flask/app.py", line 902, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/.venv/lib/python3.12/site-packages/flask_restful/__init__.py", line 489, in wrapper
    resp = resource(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/.venv/lib/python3.12/site-packages/flask/views.py", line 110, in view
    return current_app.ensure_sync(self.dispatch_request)(**kwargs)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/.venv/lib/python3.12/site-packages/flask_restful/__init__.py", line 604, in dispatch_request
    resp = meth(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/controllers/console/wraps.py", line 142, in decorated
    return view(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/libs/login.py", line 92, in decorated_view
    return current_app.ensure_sync(func)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/controllers/console/wraps.py", line 26, in decorated
    return view(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/controllers/console/workspace/models.py", line 360, in get
    models = model_provider_service.get_models_by_model_type(tenant_id=tenant_id, model_type=model_type)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/services/model_provider_service.py", line 284, in get_models_by_model_type
    provider_configurations = self.provider_manager.get_configurations(tenant_id)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/provider_manager.py", line 131, in get_configurations
    custom_configuration = self._to_custom_configuration(
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/provider_manager.py", line 586, in _to_custom_configuration
    self.decoding_rsa_key, self.decoding_cipher_rsa = encrypter.get_decrypt_decoding(tenant_id)
                                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/helper/encrypter.py", line 35, in get_decrypt_decoding
    return rsa.get_decrypt_decoding(tenant_id)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/libs/rsa.py", line 57, in get_decrypt_decoding
    raise PrivkeyNotFoundError("Private key not found, tenant_id: {tenant_id}".format(tenant_id=tenant_id))
libs.rsa.PrivkeyNotFoundError: Private key not found, tenant_id: 98eca97f-0d92-4ddd-9b9f-4174eb947f22

chenzikun · 2024-12-16T14:23:14Z

to fix the bug, I have change the code to
case StorageType.S3:
from extensions.storage.aws_s3_storage import AwsS3Storage
return AwsS3Storage

taverngoal · 2024-12-16T14:41:14Z

me too

laipz8200 · 2024-12-16T16:40:08Z

OpenDAL can also connect to S3, and we plan to use it to support more storage types in the future.

Would you be willing to share your S3 configuration after masking sensitive information? This could help us with testing and troubleshooting issues.

chenzikun · 2024-12-17T00:29:31Z

@laipz8200

# S3 Configuration
# Whether to use AWS managed IAM roles for authenticating with the S3 service.
# If set to false, the access key and secret key must be provided.
S3_USE_AWS_MANAGED_IAM=true
# The endpoint of the S3 service.
S3_ENDPOINT=
# The region of the S3 service.
S3_REGION=cn-northwest-1
# The name of the S3 bucket to use for storing files.
S3_BUCKET_NAME=llm-dify
# The access key to use for authenticating with the S3 service.
S3_ACCESS_KEY=
# The secret key to use for authenticating with the S3 service.
S3_SECRET_KEY=
~~~~~~~~~~~~~~~~~~~~

laipz8200 · 2024-12-17T03:15:56Z

Thanks, will fix it in the next version.

majian159 · 2024-12-17T03:57:01Z

Thanks, will fix it in the next version.

Is there a temporary solution?

chenzikun · 2024-12-17T06:04:51Z

@majian159
使用旧代码，docker挂载进去，给你个参考
volumes:
- ./ext_storage.py:/app/api/extensions/ext_storage.py

laipz8200 · 2024-12-17T06:23:21Z

Thanks, will fix it in the next version.

Is there a temporary solution?

After our tests, connect to S3 with AK/SK is work well. Do you use AWS managed IAM to access S3?

majian159 · 2024-12-17T08:28:48Z

Thanks, will fix it in the next version.

Is there a temporary solution?

After our tests, connect to S3 with AK/SK is work well. Do you use AWS managed IAM to access S3?

My exception message is:

opendal.ConfigInvalidError: ConfigInvalid (permanent) at Builder::build, context: { service: s3 } => region is missing. Please find it by S3::detect_region() or set them in env.

I use minio

It seems that the region is not configured. Is there any recommended configuration?

laipz8200 · 2024-12-17T08:41:25Z

Thanks, will fix it in the next version.

Is there a temporary solution?

After our tests, connect to S3 with AK/SK is work well. Do you use AWS managed IAM to access S3?

My exception message is:
opendal.ConfigInvalidError: ConfigInvalid (permanent) at Builder::build, context: { service: s3 } => region is missing. Please find it by S3::detect_region() or set them in env.
I use minio

It seems that the region is not configured. Is there any recommended configuration?

Try S3_REGION=us-east-1

laipz8200 · 2024-12-17T13:06:45Z

Hi @chenzikun, I’ve brought back the boto3-based S3 Storage in #11754, and it should now work seamlessly without any configuration changes. Feel free to give it a try!

Also, the updated OpenDAL now supports IAM and offers better performance and retry capabilities. You can update the configuration to start using it.

dosubot bot added the 🐞 bug Something isn't working label Dec 16, 2024

crazywoola assigned laipz8200 Dec 16, 2024

laipz8200 mentioned this issue Dec 17, 2024

feat: full support for opendal and sync configurations between .env and docker-compose #11754

Merged

5 tasks

crazywoola closed this as completed in #11754 Dec 18, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[bug]upload file error #11710

[bug]upload file error #11710

chenzikun commented Dec 16, 2024 •

edited by crazywoola

Loading

chenzikun commented Dec 16, 2024

dosubot bot commented Dec 16, 2024

chenzikun commented Dec 16, 2024

dosubot bot commented Dec 16, 2024

chenzikun commented Dec 16, 2024

chenzikun commented Dec 16, 2024

taverngoal commented Dec 16, 2024

laipz8200 commented Dec 16, 2024

chenzikun commented Dec 17, 2024

laipz8200 commented Dec 17, 2024

majian159 commented Dec 17, 2024

chenzikun commented Dec 17, 2024

laipz8200 commented Dec 17, 2024

majian159 commented Dec 17, 2024

laipz8200 commented Dec 17, 2024

laipz8200 commented Dec 17, 2024

[bug]upload file error #11710

[bug]upload file error #11710

Comments

chenzikun commented Dec 16, 2024 • edited by crazywoola Loading

Self Checks

Dify version

Cloud or Self Hosted

Steps to reproduce

✔️ Expected Behavior

❌ Actual Behavior

chenzikun commented Dec 16, 2024

dosubot bot commented Dec 16, 2024

chenzikun commented Dec 16, 2024

dosubot bot commented Dec 16, 2024

chenzikun commented Dec 16, 2024

chenzikun commented Dec 16, 2024

taverngoal commented Dec 16, 2024

laipz8200 commented Dec 16, 2024

chenzikun commented Dec 17, 2024

laipz8200 commented Dec 17, 2024

majian159 commented Dec 17, 2024

chenzikun commented Dec 17, 2024

laipz8200 commented Dec 17, 2024

majian159 commented Dec 17, 2024

laipz8200 commented Dec 17, 2024

laipz8200 commented Dec 17, 2024

chenzikun commented Dec 16, 2024 •

edited by crazywoola

Loading