Understanding SimilarityEvaluation and Similarity Threshold

[mayalinetsky-kryon]

I'm really confused about how the similarity evaluation function works and where the threshold takes effect.

Context

In my code, I initialize a semantic cache like so:

cache_base = CacheBase("sqlite")
    vector_base = VectorBase("faiss", dimension=embedding_onnx.dimension)
    data_manager = get_data_manager(cache_base, vector_base, max_size=100000)
    cache.init(
        embedding_func=embedding_onnx.to_embeddings,
        data_manager=data_manager,
        similarity_evaluation=WrapEvaluation(),
        config=Config(similarity_threshold=0.6),
    )

Where WrapEvaluation is just a wrap of the SearchDistanceEvaluation class for easy debugging (with max_distance set to 1):

class WrapEvaluation(SearchDistanceEvaluation):

        def __init__(self):
            super().__init__(positive=False, max_distance=1)

        def evaluation(self, src_dict, cache_dict, **kwargs):
            rank1 = super().evaluation(src_dict, cache_dict, **kwargs)
            return rank1

According to the documentation on SearchDistanceEvaluation, when positive is set to False, that means that the bigger the distance that was calculated in the retrieval stage (search_result), the less similar the two queries.
But, rank1, the output of WrapEvaluation.evaluation is exactly the opposite - the higher it is the higher the similarity:

Example 1:
Cache question: "How do I generate a pptx?"
search_result: (0.41, 1)
Question: "What is the process for generating a pptx?"
rank1: 0.59

Example 2:
Cache question: "How do I generate a pptx?"
search_result: (0.28, 1)
Question: "How do I generate a pptx file?"
rank1: 0.72

First Question:

What does "rank1" mean?

Second Question:

To which value does the similarity threshold refer to?
Meaning we want to get a hit if x>=threshold, what is x? rank1 or search_result?
My code has a threshold of 0.6 and I got a hit for rank1=0.72 but miss for rank1=0.8.

Third Question :

When I set max_distance to 1 and similarity_threshold to 1, I don't get any hits. Why??

[SimFG]

This maximum distance actually depends on the VectorBase you set. For example, with the default faiss, the resulting search distance range should be 0-4. By default, the smaller the distance, the smaller the distance. At the same time, the range of similarity_threshold is 0-1.

For example, if the similarity_threshold is now set to 0.6, the distance obtained by using faiss between the two vectors is now 0.3. At this time, the smaller the similarity distance, the more similar they are. So the similarity value we get is: 4-0.3=3.7, and the smallest similarity value is 0.6*4=2.4. Because 3.7 is greater than 2.4, we judge that the current cache value is valid.