Implementation of hedgehog transformer which is a Linear Attention Mechanism inspired by the concept described in "The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicry."
https://arxiv.org/pdf/2402.04347.pdf
Implementation of hedgehog transformer which is a Linear Attention Mechanism inspired by the concept described in "The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicry."
https://arxiv.org/pdf/2402.04347.pdf