Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Fix] Raise ValueError when llm-eval gets an unknown dataset to evaluate on (instead of failing silently) #1625

Merged
merged 4 commits into from
Mar 6, 2024

Conversation

dbogunowicz
Copy link
Contributor

@dbogunowicz dbogunowicz commented Feb 27, 2024

Feature Description

Fail more verbosely when passing datasets name that are not recognized by lm-evaluation-harness

from deepsparse import evaluate
stub = "zoo:opt-1.3b-opt_pretrain-quantW8A8"
datasets = ["openai_humaneval"]
res = evaluate(stub, datasets=datasets, integration="lm-eval-harness")
DeepSparse, Copyright 2021-present / Neuralmagic, Inc. version: 1.7.0.20240220 COMMUNITY | (8a843a2e) (release) (optimized) (system=avx2, binary=avx2)
[7f4e5f5ed640 >WARN<  operator() ./src/include/wand/utility/warnings.hpp:14] Generating emulated code for quantized (INT8) operations since no VNNI instructions were detected. Set NM_FAST_VNNI_EMULATION=1 to increase performance at the expense of accuracy.
Traceback (most recent call last):
  File "/nm/drive0/damian/deepsparse/hehe.py", line 4, in <module>
    res = evaluate(stub, datasets=datasets, integration="lm-eval-harness")
  File "/nm/drive0/damian/deepsparse/src/deepsparse/evaluation/evaluator.py", line 65, in evaluate
    return eval_integration(
  File "/nm/drive0/damian/deepsparse/src/deepsparse/evaluation/integrations/lm_evaluation_harness.py", line 75, in integration_eval
    raise ValueError(
ValueError: could recognize the dataset: openai_humaneval. Make sure that the requested dataset is compatible with the llm-evaluation-harness

@dbogunowicz dbogunowicz merged commit 14b59a8 into main Mar 6, 2024
5 checks passed
@dbogunowicz dbogunowicz deleted the fix/damian/missing_dataset branch March 6, 2024 13:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants