Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Store the list of updated IDs directly in LMDB instead of a roaring bitmap #99

Merged
merged 2 commits into from
Oct 1, 2024

Conversation

irevoire
Copy link
Member

@irevoire irevoire commented Oct 1, 2024

Pull Request

After chatting with @ManyTheFish, we realized that deserializing a roaring bitmap, inserting an item, and serializing it again into LMDB every time we insert a new item wasn’t the smartest idea.
Ideally, we should write each document in a simple file or in RAM, but that adds a lot of complexity to the usage of arroy.
By simply writing the updated item ID directly in a database we were able to add 1M items in 1.7s instead of 18s+.

The drawback is that we use slightly more memory, but as a reminder, adding 150M vectors in one batch would only use around 0.5GiB (plus the LMDB internal structures).
This is clearly acceptable considering it’s memory mapped, and storing the vectors is going to take at least around 300GiB.

What does this PR do?

  • Create a new kind of node ID updated where every associated ItemId represents the item ID of something we updated
  • Update the rest of the code accordingly
  • The tests still works 👌

@irevoire irevoire added indexing Everything related to indexing performance db-breaking labels Oct 1, 2024
@irevoire irevoire added this to the v0.4.0 milestone Oct 1, 2024
Copy link
Member

@ManyTheFish ManyTheFish left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

@irevoire irevoire merged commit d72b469 into main Oct 1, 2024
8 checks passed
@irevoire irevoire deleted the store-the-updated-id-in-lmdb branch October 1, 2024 14:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
db-breaking indexing Everything related to indexing performance
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants