[Question] Int8 Gemm's perf degraded in real models.

[foreverlms]

I have encountered a problem that whe benchmark the

TensorRT-LLM/cpp/tensorrt_llm/kernels/cutlass_kernels/int8_gemm/int8_gemm_template.h

Lines 62 to 63 in a65dba7

	void genericInt8GemmKernelLauncher(int8_t const* A, int8_t const* B, tk::QuantMode quantOption, float const* alphaCol,
	float const* alphaRow, T* C, int m, int n, int k, tkc::CutlassGemmConfig gemmConfig, char* workspace,

Int8 cutlass kernel seperately, the kernel really beats the fp16 gemm kernel.
But when it comes to real model, the int8 gemm kernel's perf degraded a lot. (I did use gemmprofilerplugin).

Two pictures from the nsight:

Int8 Gemm (above: seperate kernels, below: in real models)

int8 in benchmark:

int8 in models:

Device: A100 SXM-80GB

For 16, 6144, 4096; 14us -> 24us, almost doubbled. The config is exactly the same as the nsight shows.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 15 days."

[foreverlms]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 15 days."

!