Merging LoRA weights into a quantized model

[jayavanth]

Hello, I noticed that merging LoRA weights into a quantized model is not supported. How would one go about adding this support? I can help if it's not too difficult to do. Thanks!

[hiyouga]

We are literally unable to merge LoRA weights into a GPTQ quantized model because it needs calibration datasets to quantize the model.
You could download the unquantized model and merge the LoRA weights, then use the export script to quantize the model.

casper-hansen/AutoAWQ#85 (comment)
huggingface/optimum#1287 (comment)

[arashakb]

Your answer doesn't make sense. Cause we can gather calibration dataset to quantize the model anyway (or they might be a quantized version already available with either PTQ or AWT). There should be another reason on how the quantized version is not compatible.

[hiyouga]

Quantization process is uninvertible, although we can collect calibration dataset, we cannot acquire the original model to merge lora into it, unless we download the unquantized model instead.