Implementation of Block Recurrent Transformer - Pytorch

Implementation of Flash Attention in Jax

RAN: Recurrent Attention Networks for Long-text Modeling | Findings of ACL23

TidalDecode: A Fast and Accurate LLM Decoding with Position Persistent Sparse Attention

A repository to get train transformers to access longer context for causal language models, most of these methods are still in testing. Try them out if you'd like but please lmk your results so we don't duplicate work :)