What does "weights_scaling_factor_2" mean in safetensor results of awq_w4a8

[gujiewen]

I follow this step to do quantization for qwen2 model.
Then I got the safetensor results like

What does ' prequant_scaling_factor', 'activation_scaling_factor', 'weights_scaling_factor', 'weights_scaling_factor_2' mean. And how are they used in the w4a8 gemm?

[Barry-Delaney]

For a linear layer with GEMM shape [M, N, K], we need these components in the TRT-LLM layer:

Name	Dtype	Shape	Layout
{LAYER_NAME}.weight	float16	[K, N / 4]	Interleaved and packed INT4
{LAYER_NAME}.weight_scaling_factor	float16	[K / group_size, N]	Row-major
{LAYER_NAME}.activation_scaling_factor	float16	[K]	Row-major
{LAYER_NAME}.alpha	float32	[1]	-

The calculation process is:
output = FP16(FP8(act * activation_scaling_factor) * FP8(weight * weight_scaling_factor) * alpha)

However, the checkpoint will have more parameters, here is how they are converted when building the engine.

[gujiewen]

For a linear layer with GEMM shape [M, N, K], we need these components in the TRT-LLM layer:

Name Dtype Shape Layout
{LAYER_NAME}.weight float16 [K, N / 4] Interleaved and packed INT4
{LAYER_NAME}.weight_scaling_factor float16 [K / group_size, N] Row-major
{LAYER_NAME}.activation_scaling_factor float16 [K] Row-major
{LAYER_NAME}.alpha float32 [1] -
The calculation process is: output = FP16(FP8(act * activation_scaling_factor) * FP8(weight * weight_scaling_factor) * alpha)

However, the checkpoint will have more parameters, here is how they are converted when building the engine.

Thanks for your reply. However, in w4a8_awq, I found prequant_scaling_factor has shape of [K]. According to the source code in modeling_utils.py

        if quant_algo == QuantAlgo.W4A8_AWQ:
            for name in list(weights):
                if name.endswith('weights_scaling_factor'):
                    activation_scaling_factor = weights.pop(
                        name.replace('weights_scaling_factor',
                                     'activation_scaling_factor'))
                    weights_scaling_factor_2 = weights.pop(
                        name.replace('weights_scaling_factor',
                                     'weights_scaling_factor_2'))
                    weights[name] /= weights_scaling_factor_2
                    weights[name] = weights[name].to(torch.float16).view(
                        str_dtype_to_torch(model_config.dtype))
                    weights[name.replace(
                        'weights_scaling_factor',
                        'prequant_scaling_factor')] /= activation_scaling_factor
                    weights[name.replace(
                        'weights_scaling_factor', 'alpha'
                    )] = activation_scaling_factor * weights_scaling_factor_2

,alpha semms to be computed as activation_scaling_factor * weight_scaling_factor_2.

So, the calculation process of w4a8 is
output = FP16(FP8(act * prequant_scaling_factor / activation_scaling_factor) * FP8(weight * weight_scaling_factor / weight_scaling_factor_2) * activation_scaling_factor * weight_scaling_factor_2),
If we set
activation_scaling_factor'= prequant_scaling_factor / activation_scaling_factor
and
weight_scaling_factor'=weight_scaling_factor / weight_scaling_factor_2
the formula becomes
output = FP16(FP8(act * activation_scaling_factor') * FP8(weight * weight_scaling_factor') * alpha) which is your form.

Am I right?

[Barry-Delaney]

Exactly.
For clearer understanding, you can consider W4A8_AWQ as W4A16_AWQ + FP8.
In addition to the components of W4A16_AWQ, i.e., prequant_scaling_factor, weight_scaling_factor, FP8 will provide 2 more per-tensor scaling factors activation_scaling_factor and weight_scaling_factor_2. In order to having them combined in one GEMM, we have:

• Multiplied the per-tensor activation_scaling_factor into prequant_scaling_factor
• Multiplied the per-tensor weight_scaling_factor_2 into weight_scaling_factor
• Exposed alpha as layer parameters for de-quantization