How to use Langchain's Indexing API with Qdrant Vectorstore to avoid duplicate and unchanged content?

[myhendry]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

collection_name = 'docs1'
model = ChatOpenAI(model='gpt-3.5-turbo', api_key=os.getenv("OPENAI_API_KEY"))
client = QdrantClient(url='https://dummy-url.gcp.cloud.qdrant.io', api_key=os.getenv('QDRANT_VECTOR_DB_API_KEY'))
# client.create_payload_index(collection_name, field_name='page_content', field_schema='keyword')
embeddings = OpenAIEmbeddings(model='text-embedding-3-small')
vector_store = QdrantVectorStore(client=client, embedding=embeddings, collection_name='docs1')

Description

Hi. How should i use langchain's indexing API with qdrant vectorstore? In langchain's documentation, there is a segment on langchain's docs on how to use langchain's indexing API with Elasticsearchstore. Langchain Indexing API with Elasticsearchstore. however, as the qdrantvectorstore parameters are different, i can't figure out how to make langchain's indexing API work with QdrantVectorstore

System Info

aiohappyeyeballs 2.4.4 Happy Eyeballs for asyncio
aiohttp 3.11.11 Async http client/server framework (asyncio)
aiosignal 1.3.2 aiosignal: a list of registered asynchronous callbacks
annotated-types 0.7.0 Reusable constraint types to use with typing.Annotated
anyio 4.7.0 High level compatibility layer for multiple asynchronous event lo...
attrs 24.3.0 Classes Without Boilerplate
certifi 2024.12.14 Python package for providing Mozilla's CA Bundle.
charset-normalizer 3.4.0 The Real First Universal Charset Detector. Open, modern and activ...
click 8.1.7 Composable command line interface toolkit
dataclasses-json 0.6.7 Easily serialize dataclasses to and from JSON.
distro 1.9.0 Distro - an OS platform information API
dnspython 2.7.0 DNS toolkit
email-validator 2.2.0 A robust email address syntax and deliverability validation library.
fastapi 0.115.6 FastAPI framework, high performance, easy to learn, fast to code,...
fastapi-cli 0.0.7 Run and manage FastAPI apps from the command line with FastAPI CL...
frozenlist 1.5.0 A list-like structure which implements collections.abc.MutableSeq...
grpcio 1.68.1 HTTP/2-based RPC framework
grpcio-tools 1.68.1 Protobuf code generator for gRPC
h11 0.14.0 A pure-Python, bring-your-own-I/O implementation of HTTP/1.1
h2 4.1.0 HTTP/2 State-Machine based protocol implementation
hpack 4.0.0 Pure-Python HPACK header compression
httpcore 1.0.7 A minimal low-level HTTP client.
httptools 0.6.4 A collection of framework independent HTTP protocol utils.
httpx 0.28.1 The next generation HTTP client.
httpx-sse 0.4.0 Consume Server-Sent Event (SSE) messages with HTTPX.
hyperframe 6.0.1 HTTP/2 framing layer for Python
idna 3.10 Internationalized Domain Names in Applications (IDNA)
jinja2 3.1.4 A very fast and expressive template engine.
jiter 0.8.2 Fast iterable JSON parser.
jsonpatch 1.33 Apply JSON-Patches (RFC 6902)
jsonpointer 3.0.0 Identify specific nodes in a JSON document (RFC 6901)
langchain 0.3.13 Building applications with LLMs through composability
langchain-community 0.3.13 Community contributed LangChain integrations.
langchain-core 0.3.28 Building applications with LLMs through composability
langchain-openai 0.2.14 An integration package connecting OpenAI and LangChain
langchain-qdrant 0.2.0 An integration package connecting Qdrant and LangChain
langchain-text-splitters 0.3.4 LangChain text splitting utilities
langsmith 0.2.4 Client library to connect to the LangSmith LLM Tracing and Evalua...
markdown-it-py 3.0.0 Python port of markdown-it. Markdown parsing, done right!
markupsafe 3.0.2 Safely add untrusted strings to HTML/XML markup.
marshmallow 3.23.2 A lightweight library for converting complex datatypes to and fro...
mdurl 0.1.2 Markdown URL utilities
multidict 6.1.0 multidict implementation
mypy-extensions 1.0.0 Type system extensions for programs checked with the mypy type ch...
numpy 2.2.0 Fundamental package for array computing in Python
openai 1.58.1 The official Python library for the openai API
orjson 3.10.12 Fast, correct Python JSON library supporting dataclasses, datetim...
packaging 24.2 Core utilities for Python packages
portalocker 2.10.1 Wraps the portalocker recipe for easy usage
propcache 0.2.1 Accelerated property cache
protobuf 5.29.2
pydantic 2.10.4 Data validation using Python type hints
pydantic-core 2.27.2 Core functionality for Pydantic validation and serialization
pydantic-settings 2.7.0 Settings management using Pydantic
pygments 2.18.0 Pygments is a syntax highlighting package written in Python.
python-dotenv 1.0.1 Read key-value pairs from a .env file and set them as environment...
python-multipart 0.0.20 A streaming multipart parser for Python
pyyaml 6.0.2 YAML parser and emitter for Python
qdrant-client 1.12.1 Client library for the Qdrant vector search engine
regex 2024.11.6 Alternative regular expression module, to replace re.
requests 2.32.3 Python HTTP for Humans.
requests-toolbelt 1.0.0 A utility belt for advanced users of python-requests
rich 13.9.4 Render rich text, tables, progress bars, syntax highlighting, mar...
rich-toolkit 0.12.0 Rich toolkit for building command-line applications
setuptools 75.6.0 Easily download, build, install, upgrade, and uninstall Python pa...
shellingham 1.5.4 Tool to Detect Surrounding Shell
sniffio 1.3.1 Sniff out which async library your code is running under
sqlalchemy 2.0.36 Database Abstraction Library
starlette 0.41.3 The little ASGI library that shines.
tenacity 9.0.0 Retry code until it succeeds
tiktoken 0.8.0 tiktoken is a fast BPE tokeniser for use with OpenAI's models
tqdm 4.67.1 Fast, Extensible Progress Meter
typer 0.15.1 Typer, build great CLIs. Easy to code. Based on Python type hints.
typing-extensions 4.12.2 Backported and Experimental Type Hints for Python 3.8+
typing-inspect 0.9.0 Runtime inspection utilities for typing module.
urllib3 2.2.3 HTTP library with thread-safe connection pooling, file post, and ...
uvicorn 0.34.0 The lightning-fast ASGI server.
uvloop 0.21.0 Fast implementation of asyncio event loop on top of libuv
watchfiles 1.0.3 Simple, modern and high performance file watching and code reload...
websockets 14.1 An implementation of the WebSocket Protocol (RFC 6455 & 7692)
yarl 1.18.3 Yet another URL library

[dosubot[bot]]

Hello @myhendry! I'm here to help you with your coding questions and bug fixes. Let's tackle this challenge together!

To use LangChain's Indexing API with Qdrant Vectorstore and avoid duplicate and unchanged content, you need to ensure that Qdrant supports the necessary operations required by the Indexing API. Specifically, the vectorstore must support:

1. Document Addition by ID: Use the addDocuments method with an ids argument to add documents by ID. However, it's unclear if QdrantVectorStore supports this method directly, so you may need to verify its compatibility or consider using a supported vectorstore.

2. Deletion by ID: The QdrantVectorStore supports the delete method with an ids argument for deleting documents by ID. You can use it like this:

   await vectorStore.delete({ ids: ["id1", "id2"] });

The Indexing API uses a RecordManager to track document writes and avoid duplicating or rewriting unchanged content. It computes hashes for each document and stores the document hash, write time, and source ID in the record manager. This allows the API to determine if content has changed and manage deletions accordingly.

You can choose between different deletion modes (None, Incremental, Full) to manage how old content is cleaned up:

• None: No automatic cleanup.
• Incremental: Continuously cleans up mutations of source and derived documents.
• Full: Cleans up at the end of indexing, including deleted source documents.

Ensure that Qdrant is compatible with these requirements to effectively use the Indexing API for avoiding duplicate and unchanged content. If Qdrant is not listed as a compatible vectorstore, you may need to verify its compatibility or consider using a supported vectorstore like PGVector, Chroma, or Pinecone ^[1].

For initializing a QdrantVectorStore, you need to set up the Qdrant URL and API key, specify the collection name, and configure the embeddings. Here's a basic setup:

const embeddings = new OpenAIEmbeddings({
  model: "text-embedding-3-small",
});

const vectorStore = await QdrantVectorStore.fromExistingCollection(embeddings, {
  url: process.env.QDRANT_URL,
  apiKey: process.env.QDRANT_API_KEY,
  collectionName: "your_collection_name",
});

Make sure to set the QDRANT_URL and QDRANT_API_KEY environment variables with your Qdrant instance details ^[2].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}