ValueError: Only one class present in y_true. ROC AUC score is not defined in that case.

[gburachas]

Hi,
I'm excited to try out your LLM uncertainty estimation method. Thanks for putting it out!
I've been going over the Readme example. I made it work (although as is the code hangs on the openai call with a defunct hardwired OAI API key). I was able to walk through the demo up to step 5:
python tools/compute_metrics_ambigqa.py
and then get the error:
Traceback (most recent call last):
File "tools/compute_metrics_gsm8k.py", line 38, in
auroc = roc_auc_score(refined_labels, -xs_array)
File "/home/giedrius/anaconda3/envs/LLMuncertainty/lib/python3.8/site-packages/sklearn/utils/_param_validation.py", line 211, in wrapper
return func(*args, **kwargs)
File "/home/giedrius/anaconda3/envs/LLMuncertainty/lib/python3.8/site-packages/sklearn/metrics/_ranking.py", line 626, in roc_auc_score
return _average_binary_score(
File "/home/giedrius/anaconda3/envs/LLMuncertainty/lib/python3.8/site-packages/sklearn/metrics/_base.py", line 75, in _average_binary_score
return binary_metric(y_true, y_score, sample_weight=sample_weight)
File "/home/giedrius/anaconda3/envs/LLMuncertainty/lib/python3.8/site-packages/sklearn/metrics/_ranking.py", line 381, in binary_roc_auc_score
raise ValueError(
ValueError: Only one class present in y_true. ROC AUC score is not defined in that case.
(LLMuncertainty) giedrius@lamda1:~/Projects/LLMuncertainty/llm_uncertainty$ python evaluate

I also get the same error when executing the scripts:
scripts/nq_pipeline.sh
scripts/gsm8k_pipeline.sh
but keep getting the same error.

Would love to see this resolved!
thanks-
gtb

[hbr690188270]

Hi,

Thank you so much for your interest in our work. I am sorry for the delayed reply (not sure why github did not send me notifications about new issues). Also the ICML deadlines have kept me quite occupied. I have recently addressed some bugs that were causing execution errors. Could you please take a moment to check if your issue has been resolved?

Best,
Bairu