Skip to content

Commit

Permalink
Improve efficiency in factual correctness for precision mode (#1578)
Browse files Browse the repository at this point in the history
### Description

This pull request optimizes the `_single_turn_ascore` function by
conditionally skipping unnecessary computations when the mode is set to
"precision." Previously, the function calculated `response_reference`
regardless of the scoring mode, which consumed resources and potentially
increased processing costs. By avoiding the `response_reference`
calculation in "precision" mode, this update aims to enhance speed and
reduce computational expenses.

### Changes Made

- Modified `_single_turn_ascore` function:
- **Conditionally calculated `response_reference`**: The
`response_reference` is now computed only when the mode is not
"precision."

---------

Co-authored-by: jeff yang <jeffyang@jeffs-MacBook-Pro.local>
  • Loading branch information
Jeff-67 and jeff yang authored Oct 25, 2024
1 parent 2319dce commit fd5e805
Showing 1 changed file with 12 additions and 4 deletions.
16 changes: 12 additions & 4 deletions src/ragas/metrics/_factual_correctness.py
Original file line number Diff line number Diff line change
Expand Up @@ -244,13 +244,21 @@ async def _single_turn_ascore(
reference_response = await self.verify_claims(
premise=reference, hypothesis_list=response_claims, callbacks=callbacks
)
response_reference = await self.verify_claims(
premise=response, hypothesis_list=reference_claims, callbacks=callbacks
)


if self.mode != "precision":
response_reference = await self.verify_claims(
premise=response, hypothesis_list=reference_claims, callbacks=callbacks
)
else:
response_reference = np.array([])

true_positives = sum(reference_response)
false_positives = sum(~reference_response)
false_negatives = sum(~response_reference)
if self.mode != "precision":
false_negatives = sum(~response_reference)
else:
false_negatives = 0

if self.mode == "precision":
score = true_positives / (true_positives + false_positives + 1e-8)
Expand Down

0 comments on commit fd5e805

Please sign in to comment.