Why the inference always fallback to CPU?

[magic282]

Hi all,

Thanks for the great easy-to-use tool. I am facing an issue that the inference will fallback to CPU and not using GPU. Here is an screenshot:

My machine has 8*80GA100, and my command is:

HF_DATASETS_OFFLINE=1 TRANSFORMERS_OFFLINE=1 python run.py --datasets ceval_ppl --hf-path /workspace/data/open/baichuan/Baichuan2-7B-Base  --tokenizer-path /workspace/data/open/baichuan/Baichuan2-7B-Base  --tokenizer-kwargs padding_side='left' truncation='left' trust_remote_code=True  --model-kwargs device_map='auto' trust_remote_code=True   --max-seq-len 2048  --max-out-len 100  --batch-size 1  --no-batch-padding --num-gpus 1 --max-partition-size 512

Any help or hint is appreciated.

[yingfhu]

Your infer tasks were on GPUs already,

The cpu tasks are for evalution which does not involves any GPU

[magic282]

I see. Thank you for the clarification.