adding new dataset - registry error

[ycYiwei]

I'm trying to add the CValues dataset (the multiple choice set) onto OpenCompass. Following the instruction here, I created a cvalues.py file under opencompass/datasets:

import datasets
from .base import BaseDataset
from opencompass.registry import LOAD_DATASET

@LOAD_DATASET.register_module()
class CValuesDataset(BaseDataset):

    @staticmethod
    def load(path: str):

        with open(path) as f:
            data = json.load(f)
        
        rows = []
        for index, question in enumerate(data):
            row.append({"prompt": question["prompt"], label: question["label"]})

        dataset = Dataset.from_dict({
            'prompt': [row['prompt'] for row in rows],
            'label': [row['label'] for row in rows],
        })

        return dataset

I then created a config file under configs/datasets:

from opencompass.openicl.icl_prompt_template import PromptTemplate
from opencompass.openicl.icl_retriever import ZeroRetriever
from opencompass.openicl.icl_inferencer import GenInferencer
from opencompass.openicl.icl_evaluator import AccEvaluator
from opencompass.datasets import CValuesDataset
from opencompass.utils.text_postprocessors import first_capital_postprocess

cvalues_reader_cfg = dict(
    input_columns=['prompt'], output_column='label')

cvalues_infer_cfg = dict(
    prompt_template=dict(
        type=PromptTemplate,
        template=dict(round=[
            dict(
                role="HUMAN",
                prompt="{{prompt}}"),
        ])),
    retriever=dict(type=ZeroRetriever),
    inferencer=dict(type=GenInferencer))

cvalues_eval_cfg = dict(
    evaluator=dict(type=AccEvaluator),
    pred_postprocessor=dict(type=first_capital_postprocess))

cvalues_datasets = [
    dict(
        type=CValuesDataset,
        abbr='cvalues',
        path='.data/cvalues/cvalues_responsibility_mc.jsonl',
        reader_cfg=cvalues_reader_cfg,
        infer_cfg=cvalues_infer_cfg,
        eval_cfg=cvalues_eval_cfg)
]

I got the following error message when trying to run an evaluation:
KeyError: 'opencompass.datasets.CValuesDataset is not in the opencompass::load_dataset registry. Please check whether the value of opencompass.datasets.CValuesDataset is correct or it was registered as expected. More details can be found at https://mmengine.readthedocs.io/en/latest/advanced_tutorials/config.html#import-the-custom-module'

After checking out the MMEngine documentation, I added custom_imports = dict(imports=['opencompass.datasets.cvalues'], allow_failed_imports=False) in the config file after the import statements, but still get the same error. Any help regarding this problem would be appreciated!

[tonysy]

Hi, CValuses is supported here. https://github.com/InternLM/opencompass/blob/main/configs/datasets/cvalues/cvalues_responsibility_gen_4aec9f.py
Feel free to improve it if needed.

[ycYiwei]

Oh thanks! I think I cloned the repo before that update so I missed it (also it turns out my problem was mainly not importing cvalues in __init__.py)