Multimodal Embedding

[taowang1993]

Self Checks

• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
• Please do not modify this template :) and fill in all the required fields.

1. Is this request related to a challenge you're experiencing? Tell me about your story.

Currently, Dify supports only text embedding.

But I need to show users images from my documents such as diagrams and graphs.

This feature is very useful in education, medicine, legal and finance domains.

Major vector DBs already support multimodal RAG.

https://weaviate.io/blog/multimodal-models

https://milvus.io/docs/multimodal_rag_with_milvus.md

https://jina.ai/news/jina-clip-v1-a-truly-multimodal-embeddings-model-for-text-and-image/

2. Additional context or comments

No response

3. Can you help us with this feature?

• I am interested in contributing to this feature.

[friedinando]

+1

[monotykamary]

https://docs.voyageai.com/docs/multimodal-embeddings
I think this is the near future for multimodal RAG, especially since OCR for Open WebUI and now Claude's Visual PDFs are getting heavier use-cases.

[dosubot]

Hi, @taowang1993. I'm Dosu, and I'm helping the Dify team manage their backlog. I'm marking this issue as stale.

Issue Summary

• You requested the addition of multimodal embedding support in Dify, emphasizing its importance in various fields.
• @friedinando expressed support for this feature.
• @monotykamary shared documentation on multimodal embeddings and highlighted the growing relevance of multimodal RAG.

Next Steps

• Could you confirm if this issue is still relevant to the latest version of the Dify repository? If so, please comment to keep the discussion open.
• If there is no further activity, this issue will be automatically closed in 15 days.

Thank you for your understanding and contribution!