[Help] Yi-34B + Simpo + Full + Novel writing task, fine-tuning results are unsatisfactory!

[onlyfish79]

Reminder

• I have read the README and searched the existing issues.

System Info

• llamafactory version: 0.8.3.dev0
• Platform: Linux-6.5.0-41-generic-x86_64-with-glibc2.35
• Python version: 3.10.13
• PyTorch version: 2.2.1+cu121 (GPU)
• Transformers version: 4.41.2
• Datasets version: 2.18.0
• Accelerate version: 0.31.0
• PEFT version: 0.11.1
• TRL version: 0.9.4
• GPU type: NVIDIA A100-SXM4-80GB
• DeepSpeed version: 0.13.4
• Bitsandbytes version: 0.43.1
• vLLM version: 0.4.1

Reproduction

• 执行脚本

torchrun --nproc_per_node=8 --master_port=20001 src/train.py \
--stage dpo \
--pref_loss simpo \
--pref_beta 2.0 \
--simpo_gamma 1.0 \
--do_train \
--model_name_or_path /data/basemodel/Yi-34B \
--dataset novel \
--val_size 0.05 \
--dataset_dir /data/train_data \
--overwrite_cache \
--template yi \
--finetuning_type full \
--output_dir /data/dpo_model/yi34b_simpo_full \
--overwrite_output_dir \
--preprocessing_num_workers 128 \
--per_device_train_batch_size 4 \
--per_device_eval_batch_size 8 \
--gradient_accumulation_steps 4 \
--eval_strategy steps \
--eval_steps 10 \
--save_strategy steps \
--save_total_limit 1 \
--lr_scheduler_type cosine \
--logging_steps 1 \
--save_steps 230 \
--learning_rate 1e-5 \
--num_train_epochs 1.0 \
--max_grad_norm 0.5 \
--weight_decay 0 \
--warmup_ratio 0.03 \
--cutoff_len 2048 \
--plot_loss \
--bf16 \
--flash_attn fa2 \
--gradient_checkpointing \
--ddp_timeout 180000000 \
--deepspeed deepspeed/ds_zero3_offload_optimizer_param.json

• ds_zero3_offload_optimizer_param.json

{
  "bf16": {
    "enabled": "auto"
  },
  "optimizer": {
    "type": "AdamW",
    "params": {
      "lr": "auto",
      "betas": "auto",
      "eps": "auto",
      "weight_decay": "auto"
    }
  },
  "scheduler": {
    "type": "WarmupDecayLR",
    "params": {
      "total_num_steps": "auto",
      "warmup_min_lr": "auto",
      "warmup_max_lr": "auto",
      "warmup_num_steps": "auto"
    }
  },
  "zero_optimization": {
    "stage": 3,
    "offload_optimizer": {
      "device": "cpu",
      "pin_memory": true
    },
    "offload_param": {
      "device": "cpu",
      "pin_memory": true
    },
    "overlap_comm": true,
    "contiguous_gradients": true,
    "sub_group_size": 1e9,
    "reduce_bucket_size": "auto",
    "stage3_prefetch_bucket_size": "auto",
    "stage3_param_persistence_threshold": "auto",
    "stage3_max_live_parameters": 1e9,
    "stage3_max_reuse_distance": 1e9,
    "stage3_gather_16bit_weights_on_model_save": true
  },
  "gradient_accumulation_steps": "auto",
  "gradient_clipping": "auto",
  "steps_per_print": 50,
  "train_batch_size": "auto",
  "train_micro_batch_size_per_gpu": "auto",
  "wall_clock_breakdown": false
}

• 数据集字段：instruction, chosen, rejected

• 任务：小说创作，输入：提供一个小说片段，输出：2/3个选项分支

• 问题描述：

  • 环境，数据集，基础模型，全部一致，使用DPO方式，微调结果是正向的，但是使用simpo（全参和Lora都微调了），微调后的模型，生成的结果很差，结果如下：
    

• 问题： 请问是因为Simpo不适合小说创作任务吗？

Expected behavior

期望返回的结果形如下：

Others

No response

[onlyfish79]

如下是官方针对这个问题的回复，会尝试一下官方的建议，有结论在同步。

[onlyfish79]

参考官方的优化建议，修改微调参数：

结论: lr=6e-7（之前使用的是1e-5）, 针对任务：小说创作，输入：提供一个小说片段，输出：2/3个选项分支，微调后模型生成内容和格式都正确了。