Skip to content

Commit

Permalink
[Fix] Raise ValueError when llm-eval gets an unknown dataset to evalu…
Browse files Browse the repository at this point in the history
…ate on (instead of failing silently) (#1625)

* initial commit

* fixing tests
  • Loading branch information
dbogunowicz authored Mar 6, 2024
1 parent 1ae58fb commit 14b59a8
Showing 1 changed file with 11 additions and 2 deletions.
13 changes: 11 additions & 2 deletions src/deepsparse/evaluation/integrations/lm_evaluation_harness.py
Original file line number Diff line number Diff line change
Expand Up @@ -67,8 +67,17 @@ def integration_eval(
"""
pipeline = DeepSparseLM(pipeline=pipeline, batch_size=batch_size)

datasets = (",").join(datasets) if isinstance(datasets, list) else datasets
task_names = utils.pattern_match(datasets.split(","), tasks.ALL_TASKS)
datasets = datasets if isinstance(datasets, list) else [datasets]
task_names = []
for idx, dataset in enumerate(datasets):
task_name = utils.pattern_match(dataset, tasks.ALL_TASKS)
if not task_name:
raise ValueError(
f"could recognize the dataset: {datasets[idx]}. Make sure "
"that the requested dataset is compatible with the "
"llm-evaluation-harness"
)
task_names.extend(task_name)

_LOGGER.info(f"Selected Tasks: {task_names}")

Expand Down

0 comments on commit 14b59a8

Please sign in to comment.