NullStack Challenge 02

Hopper tensor cores (Hopper w4a16 linear)

Design a high-throughput fused W4A16 GEMM kernel targeting NVIDIA H100 Hopper Tensor Cores.

Maximize asynchronous Tensor Memory Accelerator (TMA) and warp-group matrix multiply and accumulate (WGMMA) instructions to eliminate memory stall cycles during 4-bit weight dequantization.

Weight matrix W is stored in packed 4-bit format with FP8/FP16 group scales.

Activation matrix A is stored in FP16 row-major layout.

Accumulate results in FP32 and store output in FP16 format.

Interface Signals

Input: Tensor descriptor for Matrix A (FP16)
Input: Tensor descriptor for Matrix W (Packed INT4/FP4)
Input: Scale & bias vectors
Output: GEMM result tensor (FP16)
Config: Tile dimensions (M=128, N=256, K=64)

Execution Sequence (multiply_en asserted)

Zero global memory spills across thread blocks.
Overlapped TMA loading with asynchronous pipeline stages (depth >= 4).
Achieve >85% of theoretical peak FP16 Tensor Core FLOPS.

Design Requirements

Implement the kernel using clean CUDA C++ / PTX with Hopper WGMMA intrinsics.
Provide reproducible benchmark harness with correctness verification against cuBLAS.