Conditional Upserts (OR allowing specify a "version" field, search will return results with largest "version" field value)

[xiaobingxia-at]

Hey team, I need to do a conditional upserts because of multiple threads could update the same vector at the same time. Let's say we only update the specific record if the input record's some column value is larger than the existing value in Milvus.

Or can we specify a column as "version" column, users can just do the upserts along with the "version" info. But when doing the search, only the records with "largest version value" will be returned.

Can we support this feature?
Thanks!

[yhmo]

Each upsert action will generate a new entity, the old entity will be marked as deleted. Search will return the new entity since the old one is marked as deleted. No need to add a version field.

[xiaobingxia-at]

To be clear, I'm talking about a racing condition here.
let's say worker thread 1 is inserting an entity at time t1, worker thread 2 is inserting an entity at time t2. t1 < t2, meaning worker thread 1 goes first. The ideal result is we see the entity inserted by worker thread 2 since it replaces the entity inserted by thread 1.

But somehow worker thread 1's insert requests comes late because of the networking delay. In the end, we will see thread 1's entity replacing the thread 2's entity, which is not correct.

[yhmo]

• If the two workers insert two entities with different ID
  You can add a field "version"(int type) to the collection. Each request set this version number by timestamp. Search request returns topk items, you can set output_fields=["version"] so that it returns the values of "version", then you can compare the version values by yourself.

• If the two workers insert two entities with the same ID
  The search result might contain any of the two entities, sometimes is the t1 entity, sometimes is the t2 entity, depends on the filtering expression. If you did set the filtering expression, the t2 entity will be returned.
  By the way, inserting two entities with the same ID is an incorrect client-side action, milvus has no ability to deduce duplicated id items for insert().

[xiaofan-luan]

right now this has to been done with the help of another database.

We can probably take this into consideration on 3.0, when PK index is supported