rfcs: graph: support int4/int8 compression for K/V in fused SDPA

[wzt1997]

Description

This is to propose to support int4/int8 compression for K/V in fused SDPA.
Link to the rendered document.

[dzarukin]

Not sure how that matches the case when Batch dimension should be broadcasted for scales/zero-points:

W = [B, K, N]
pre-scale W: [1, gK, N] x [B, K, N] = W'
matmul: [B, M, K] x W' = [B, M, N]

Use case I've seen are like that, batch dimension doesn't receive its own dimension of scales.

[wzt1997]

Thanks for providing the case. According to potential request from IPEX and per-token quantization, we decide to update the scale/zp input shape requirement for per-group quantization for scalability and flexibility. One potential option is to allow 1 on dimensions other than the last two, K and N.

[TaoLv]

For API completeness, I feel yes we need to add both s4 and u4. Just for my curiosity, do you know if both s4 and u4 are used for this int4 K/V compression request? If both are used, any difference in quantization recipe between s4 and u4?

[wzt1997]

According to user request, they are likely to use s4 for KV storage. But as the KV compression is still WIP, it might change in the future. Regarding the difference between u4 and s4 recipes, since int4 data types always use asymmetric quantization, the parameters will be quite similar for u4 and s4. The difference should be mainly related to the de-quantization logic.