Skip to content

Issues: bigcode-project/bigcode-evaluation-harness

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

52 Open 98 Closed
52 Open 98 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

HumanEval-X generation appears to not time out called subproceses
#290 opened Nov 24, 2024 by nielstron
HumanEval-X Go evaluation raises error
#289 opened Nov 23, 2024 by nielstron
Adding an assistant_model Argument for Speculative Decoding
#288 opened Nov 23, 2024 by ilyasoulk
Could you share a completed file of generations_mbppplus.json
#287 opened Nov 13, 2024 by marybloodyzz
mbpp pass@1 is lower than mbppplus pass@1 on starcoder2?
#285 opened Nov 6, 2024 by Liuzzyg
Continuing / Extending Previous Results from Generating and Evaluating?
#284 opened Nov 1, 2024 by RylanSchaeffer
Docker image for multiple evalulation broken
#283 opened Oct 27, 2024 by Extirpater
1
[REQUEST] Apply tokenizer chat template for HumanEvalPack
#282 opened Oct 25, 2024 by timrbula
1
[REQUEST] support model-parallel evaluation for big models
#278 opened Oct 18, 2024 by tarahjjeon
trust_remote_code is not been passed to dataset
#276 opened Sep 24, 2024 by login256
1
ValueError: Infilling not yet supported for:/Meta-Llama-3.1-8B
#275 opened Sep 23, 2024 by kbmlcoding
Evaluation result of bigcode/starcoder2-3b on gsm8k_pal does not matched the paper
#272 opened Sep 13, 2024 by nongfang55
Evaluating a Model with a Local Dataset in an Offline Environment
#271 opened Sep 12, 2024 by ankush13r
8
Any pypi package for this tool?
#270 opened Sep 10, 2024 by zhimin-z
Unable to execute the MultiPL-E task for python language
#268 opened Sep 4, 2024 by manthan0227
ImportError: cannot import name 'SyncManager' from partially initialized module 'multiprocessing.managers'
#267 opened Sep 3, 2024 by xinghuang2050
1
What is fine-tuning in task submission?
#266 opened Aug 29, 2024 by zhimin-z
1
Error of testing codegeex4-all-9b model
#264 opened Aug 21, 2024 by Gumingbro
1
Humaneval and MBPP results of deepseek-6.7b-coder-instruct are lower than offical report of Deepseek team
#262 opened Aug 9, 2024 by jessyford
2
HumanEval evaluation results mismatch for Codegemma-2b
#261 opened Jul 26, 2024 by berserank
1
[Possibly system specific] Wild (12% vs 20%) run-to-run swings in multiple-cpp reported scores
#258 opened Jul 18, 2024 by alat-rights
1
Need some context for certain args for Instruct Human Eval
#256 opened Jul 18, 2024 by teknium1
2
The evaluation results are inconsistent across different GPUs
#252 opened Jul 8, 2024 by DonteFlynn
Using the humanevalpack to test the ChatGLM3 model results in an abnormal score.
#251 opened Jul 5, 2024 by burger-pb
MBPP Llama3-8B-Instruct lower pass@1 score expected
#246 opened Jun 16, 2024 by YangZhou08
1
ProTip! What’s not been updated in a month: updated:<2024-10-24.