how to save checkpoint in train_gpt_demo

[hmzo]

你好，在train_gpt_demo中没看到如何存储模型的逻辑，假如我想直接使用 model.save_pretrained 保存模型该怎么做呢？
我尝试在demo最后加上

model_to_save = get_static_torch_model(model)
model_to_save.model.save_pretrained("./tmp")

但它报错了，

AttributeError: 'Linear' object has no attribute 'bias

感谢解答

[binmakeswell]

Hi @hmzo We have updated it. Thanks.
https://github.com/hpcaitech/ColossalAI/tree/main/applications/ChatGPT#how-to-saveload-checkpoint

[JingxinLee]

人家问的是train_gpt_demo , 那里面哪里有strategy

[chensnathan]

@hmzo This issue occurs to me when I use the colossalai_gemini strategy. This is caused by the function get_static_torch_model, which is responsible for changing zero_ddp models to torch models. It does not set the bias attribute to nn.Linear.

I fix the problem by manually adding two lines after this line, then reinstall:

                setattr(torch_module, sufix_param_name, torch_param)
                # add a bias attribute here
                if isinstance(torch_module, nn.Linear) and not hasattr(torch_module, 'bias'):
                      setattr(torch_module, 'bias', None)

Hope that the above information provides insights for solving the problem.