Support TEI on AMD GPUs

[japarada]

Feature request

Are there active plans to add support for generating embedding using AMD GPUs?

Motivation

AMD and Hugging Face are currently engaged in an ongoing collaboration to ensure compatibility of Hugging Face transformers with AMD ROCm and hardware platforms. Providing support for HF TEI toolkit would add a compelling alternative for deploying and serving open-source text embeddings and sequence classification models.

Your contribution

Work with other on PRs to integrate changes to support AMD GPUs.

[OlivierDehaene]

Indeed it would be interesting to support more backends.
Adding support for AMD GPUs will happen in Candle tough, not in TEI directly.

[japarada]

@OlivierDehaene Thanks for the response. Does TEI support python backend instead of Candle? I see https://github.com/huggingface/text-embeddings-inference/tree/d05c949c1234786c15b675f4419776a417519583/backends/python? Is this python code only for the grpc server implementation?

[OlivierDehaene]

This backend is an example of how you would go about adding other backends to TEI. It is currently out-dated but could be updated to support AMD.

[japarada]

Are there any active effort support to make python backend fully functional?
What models can be run at this moment?
Thank you for all the answers.

[hvico]

Hi! Is there any WiP regarding ROCm support for TEI? Thanks!

[dcbark01]

Hello? Is there any WiP regarding ROCm support for TEI? Thanks!

I'm interested in this as well. May start working on it myself, but don't want to duplicate efforts if it's already in the works.

[fxmarty]

Hi, there is some progress in #293. Would you mind sharing which AMD GPUs you are using? Thank you!

[dcbark01]

Currently using MI250s.

[puppetm4st3r]

Hi, any news on this, is there a way to serve the models with amd gpus on TEI?

regards!