Would the current implementation of Fisher Information work out of the box with Multi-head Latent Attention

[naston]

Multi-head Latent Attention is described in DeepSeekV2. Instead of caching KV they construct KV from a low dimension latent vector C which can be cached instead. KVQuant can obviously be applied to this Attention architecture but I am wondering if it can do so out of the box, specifically in regards to Fisher Information.