[CPU]PageAttn with 4bit-quantization

[zhangYiIntel]

Details:

• Add new hint to set group_size for key/value cache
• Add grouped 4bit sym/asym quantization support for PageAttentionNode
• Add grouped quantization for U8 quantization for PageAttentionNode

Tickets:

• CVS-151586

[luo-cheng2021]

LGTM, great job!

[zhangYiIntel]

@dmitry-gorokhov Could you have a review ?

[dmitry-gorokhov]

We need to align positioning regarding these options.
We already have high-level hint for KV-cache: ov::hint::kv_cache_precision. These new options are rather fine tuning options. So I would propose the following:

1. New options shouln't be treated as hints: lets move from the namespace.
2. ov::hint::kv_cache_precision should remain major (including positioning to the user) option for KV-Cache quantization control.
3. ov::hint::kv_cache_precision (like other hints) should impact values of lower level options: ov::hint::key_cache_precision/ov::hint::value_cache_precision/ov::hint::key_cache_group_size/ov::hint::value_cache_group_size. E.g. ov::hint::kv_cache_precision == u4 will result in (u8/u4/32/32) config for lower options.
4. User will have an ability to rewrite the behavior of high-level hint by changing values for low-level properties.

cc'ed @AlexKoff88 @vladimir-paramuzov @sshlyapn @p-durandin

[AlexKoff88]

I think it looks good. Just to clarify:

• We could have ov::hint::kv_cache_precision for coarse control of KV-cache quantization parameters by default. I would deprecate it at some point (not sure what is the best time).
• ov::hint::key_cache_precision, ov::hint::value_cache_precision, ov::hint::key_cache_group_size, ov::hint::value_cache_group_size are for fine-grained control of KV-cache quantization and they have higher priority over ov::hint::kv_cache_precision if defined. ov::hint::key_cache_group_size, ov::hint::value_cache_group_size should have reasonable defaults, e.g. 32 or 64 what fits the best for runtime.
• We should be able to define any of these options via the compilation config and rt_info/runtime_options subsection of the IR.

• @yury-gorbachev, shall we discuss and approve this item?

[zhangYiIntel]

@dmitry-gorokhov If not use hint namespace, do we have a better namespace for this ?

[dmitry-gorokhov]

@zhangYiIntel just ov::key_cache_precision.
You can use ov::num_streams as an example - this is low level property which is affected by high-level hints like ov::hint::performance_mode

[dmitry-gorokhov]

Random place. We need to limit key_cache_precision and value_cache_precision supported matrix for SDPA operation. Otherwise these properties doesn't have any impact on execution now.

[zhangYiIntel]

That's sure, the current SDPA only supports U8 KV-cache. Maybe we could just the validity of properties in SDPA node.

[zhangYiIntel]

One reaming problem here is the priority of kvCachePrecision and keyCachePrecision/valueCachePrecision. Inside SDPA node, we cannot tell which one is valid since all of them have default value.

[dmitry-gorokhov]

As we discussed kvCachePrecision is a hint, while keyCachePrecision/valueCachePrecision are low-level options. Low-level options should always have higher priority.
In practice you should use keyCachePrecision/valueCachePrecision in the SDPA. keyCachePrecision/valueCachePrecision should be initialized in the Config based on kvCachePrecision if no user values provided, or based on user values. (you can check how inference_precision is initialized based on execution_mode).