Is it possible to pass a pre-computed TF-IDF matrix?

[celsofranssa]

Is it possible to pass a pre-computed TF-IDF matrix (with the shape [documents, vocabulary])?

[xhluca]

Thanks for the question! Unfortunately the BM25 algorithm differ slightly from tf-idf, so the scores you will have are different. If you absolutely want to pass something, you would need to modify the indptr, indices, and data keys of obj.scores where obj is your bm25s.BM25 instance.

Here's how it is currently being built behind the scene: https://github.com/xhluca/bm25s/blob/35036613340e2511790213a6fb988e573b1936e6/bm25s/__init__.py#L255-L270C16

[celsofranssa]

Ok. However, I think of TF-IDF as a representation of a collection of documents, commonly realized as a sparse matrix with shape (documents, vocabulary_size).

Since the BM25 score function uses terms such as:

• tf(t,d): the term frequency of term t in document d.
• N: the total number of documents.
• df(t): the document frequency of term t (number of documents containing 𝑡).

it could also be calculated from a TF-IDF matrix.

The advantage is being able to enhance/filter the TF-IDF matrix before calculating the BM25 score.

[xhluca]

Yeah there's a nice parallel! i think it should be able to add the sparse matrix to the scores variable.