使用openai的数据格式微调多模态模型，数据加载的时候会报错

[anhuaxiang]

Reminder

• I have read the README and searched the existing issues.

System Info

• llamafactory version: 0.9.2.dev0
• Platform: Linux-5.15.0-89-generic-x86_64-with-glibc2.35
• Python version: 3.11.9
• PyTorch version: 2.4.0 (GPU)
• Transformers version: 4.45.2
• Datasets version: 2.21.0
• Accelerate version: 0.34.2
• PEFT version: 0.12.0
• TRL version: 0.9.6
• GPU type: NVIDIA GeForce RTX 3090
• DeepSpeed version: 0.15.3

Reproduction

llamafactory-cli train vision.yaml

Expected behavior

微调 Llama-3.2-11B-Vision-Instruct 模型时
使用openai格式数据，如
{"messages": [{"role": "system", "content": "You are an assistant that identifies uncommon cheeses."}, {"role": "user", "content": "What is this cheese?"}, {"role": "user", "content": [{"type": "image_url", "image_url": {"url": "https://upload.wikimedia.org/wikipedia/commons/3/36/Danbo_Cheese.jpg"}}]}, {"role": "assistant", "content": "Danbo"}]}

加载数据会报错
pyarrow.lib.ArrowInvalid: JSON parse error: Column(/messages/[]/content) changed from string to array in row 0 或者
ValueError: Failed to convert pandas DataFrame to Arrow Table from file /workspace/text.jsonl.
我的panda版本已经>2.0了
pandas==2.3.0

Others

如何支持openai这种格式呢？

[hiyouga]

https://github.com/hiyouga/LLaMA-Factory/tree/main/data#multimodal-image-dataset-1