can index use flash attention or xformers?

[Koishi-Star]

i want to know if it can use flash attention.thanks.

[mayokaze]

During our actual training process, we employed the flash attention2 mechanism; however, due to its heavy reliance on numerous other dependencies, the training codebase is not conducive to being open-sourced. For those seeking enhanced inference performance, we strongly advise adopting our newly released gguf quantization version or exploring alternative open-source options, including vllm and Tensor RT, among others.

[Koishi-Star]

Well, I will try vllm.Thanks.

[jyweky]

Are you planning to implement HF Transformers' flash attention2 inference in the future?