Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add option to select ONNXRuntime Execution Providers for fastembed #1119

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

Fannovel16
Copy link

@Fannovel16 Fannovel16 commented Oct 1, 2024

Related Issues

  • In some cases, fastembed does not use GPU despite onnxruntime-gpu and fastembed-gpu are installed correctly

Proposed Changes:

  • Add option onnx_providers to set ONNXRuntime Execution Providers manually.

How did you test it?

  • WIP

Notes for the reviewer

Checklist

@Fannovel16 Fannovel16 requested a review from a team as a code owner October 1, 2024 12:16
@Fannovel16 Fannovel16 requested review from shadeMe and removed request for a team October 1, 2024 12:16
@CLAassistant
Copy link

CLAassistant commented Oct 1, 2024

CLA assistant check
All committers have signed the CLA.

@github-actions github-actions bot added integration:fastembed type:documentation Improvements or additions to documentation labels Oct 1, 2024
@anakin87
Copy link
Member

anakin87 commented Oct 3, 2024

Just a quick thought: I don't think this would work, because this integration explicitly depends on fastembed.
GPU support requires fastembed-gpu, which cannot be installed in the same environment as fastembed (FastEmbed docs).

Maybe I'm wrong, but I think the whole integration needs to be reworked to support GPU. There may be several ways to do that...

Related: #905

@Fannovel16
Copy link
Author

Fannovel16 commented Oct 4, 2024

Just a quick thought: I don't think this would work, because this integration explicitly depends on fastembed. GPU support requires fastembed-gpu, which cannot be installed in the same environment as fastembed (FastEmbed docs).

Maybe I'm wrong, but I think the whole integration needs to be reworked to support GPU. There may be several ways to do that...

Related: #905

@anakin87 On Google Colab, I installed fastembed-gpu and correct onnxruntime-gpu but it does not pick up GPU by default, even if "CUDAExecutionProvider" is available. Explicily stating providers fix that. I've been using this fork on my project rn

#@title Install library
!pip install git+https://github.com/deepset-ai/haystack "datasets>=2.6.1" "sentence-transformers>=3.0.0" markdown-it-py mdit_plain
!pip install langchain-text-splitters instructor

%cd /content
!git clone https://github.com/Fannovel16/haystack-core-integrations/
%cd /content/haystack-core-integrations/integrations/fastembed
!pip install .
%cd /content/haystack-core-integrations/integrations/qdrant
!pip install .
%cd /content

!pip install "onnxruntime-gpu<=1.17" -i https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-cuda-12/pypi/simple/
!pip install fastembed-gpu

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
integration:fastembed type:documentation Improvements or additions to documentation
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants