bitseek - Flash Multi Head Latent Attention inspired by deepseek.

[chetanreddyv]

Features

• 1.58-bit weight quantization (ternary values)
• 8-bit activation quantization
• Flash Multi-head Latent Attention with causal masking
• Rotary Position Embeddings (RoPE)
• SwiGLU activation in FFN layers
• SubLN normalization
• No bias terms in linear or normalization layers
• Llama 2 tokenizer integration

Architecture Overview

1. Sequence Length: 2048
2. Model Size: ~300M parameters
3. Key Components:
   • BitLinear layers for efficient weight and activation quantization
   • Rotary Position Embeddings (RoPE) for positional encoding
   • SwiGLU activation in Feed-Forward Network (FFN) layers
   • SubLN (Sub-layer Normalization) normalization
   • No bias terms in linear or normalization layers
   • Flash Multi-head latent attention inspired by DeepSeek

I have the code for this implementation but no compute, I want to contribute this code here as open source.