Question about dequantization

[HaoWeiWang]

OS

Linux

GPU Library

CUDA 12.x

Python version

3.12

Pytorch version

2.1.1

Model

No response

Describe the bug

I would like to ask why when dequantizing, we need to subtract 1024 or 64 and then add it back.

Reproduction steps

Empty

Expected behavior

Empty

Logs

No response

Additional context

No response

Acknowledgements

• I have looked for similar issues before submitting this one.
• I understand that the developers have lives and my issue will be answered when possible.
• I understand the developers of this program are human, and I will ask my questions politely.

[turboderp]

It's to center the dequantized range around zero before scaling. You mask two 4-bit values from an 8-bit field to get w_a << 4 and w_b. The former is implicitly divided by 16 (1024/64) instead of shifting the bitfield four places. The 1024 comes about because of how FP16 values are normalized. 0x6400 maps to an FP16 value of 1.0, and 0x6400 | x maps to 1.0 + x if x is an integer between 0 and 511. 0xe400 is the same but with the sign bit set.