-
Notifications
You must be signed in to change notification settings - Fork 112
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
retrieval evaluation #57
Comments
Hi, thanks for the question. We used the appropriate evaluation framework for each dataset. We used passage titles for evaluation in both HotpotQA and 2WikiMultiHop since they are unique but for MuSiQue we used the entire passage since many of them share a title. |
thank you for your kind answer and I have another question. what is the passage in your paper? ['Teutberga', ['Teutberga( died 11 November 875) was a queen of Lotharingia by marriage to Lothair II.', "She was a daughter of Bosonid Boso the Elder and sister of Hucbert, the lay- abbot of St. Maurice's Abbey."]] So, passage means a concatenated passage ? or each sentence in same title? |
right, for 2WikiMultiHop, we concatenate the sentences to make a passage and determine passage relevance by whether it has a supporting sentence within it. |
icrot_hipporag.py include a recall program.
I have question in evaluation process about below source code.
below code shows a title-level recall evaluation. (means if sp is in some title == answer title) than recall score raise.
Retrieval evaluation score is title-level in your Project?
The text was updated successfully, but these errors were encountered: