Add BERTScore as potential "non-LLM" metric / MetricWithEmbedding for context-recall and context-precision

[ahgraber]

Describe the Feature
Add BERTScore as additional evaluation metric scorer for context-precision and context-recall.

Why is the feature important for you?

As a RAGAS user trying to evaluate/benchmark my RAG applications, I would like metrics that are more deterministic than LLM-as-a-Judge and more advanced and indicative of actual recommendation performance than naive string matching. BERTscore uses embeddings to compare documents and is flexible wrt. the embedding model used. This does not require invoking LLMs (so can be run on an analyst machine locally) and provides research-validated scores that are replicable.

Additional context

• [1904.09675] BERTScore: Evaluating Text Generation with BERT
• bert_score/bert_score/score.py at master · Tiiiger/bert_score
• BERT Score - a Hugging Face Space by evaluate-metric

The BERTscore package has not been updated recently; the recommended microsoft/deberta-xlarge-mnli model only allows ~500 tokens. Models with longer context (allenai/longformer-large-4096-finetuned-triviaqa and microsoft/deberta-v3) do not perform as well, and modern embedding models (nomic-embed-*, baii/bge-*) are untested.

It would be interesting to see what happens using the same embedding model used in the RAG pipeline; that way you only have to embed the answer since you already have access the chunk embeddings from the RAG lookup.

[jjmachan]

@shahules786 what do you think? could we rework https://docs.ragas.io/en/latest/concepts/metrics/available_metrics/semantic_similarity/ for this?

[ahgraber]

I take back my suggestion re: being able to reuse embeddings -- BERTScore is calculated based on token-level embeddings.
To prevent adding a dependency on bert-score, I may just try to hack a custom Metric that calls BertScore internally.

[shahules786]

@ahgraber I have used BERTSCORE in the past, just that with the state of improvement in embeddings /LLMs I feel it doesn't make sense to adopt it anymore. It's not deterministic either. The idea of non-LLM metrics is to provide highly deterministic model-free metrics.

But if users feel adding it to ragas adds value, we can take a look at that. If not please feel free to close the issue.

[ahgraber]

@shahules786 In your experience, does [cosine] similarity over text embeddings effectively replace BERTscore?