Having issue while running the reader in Haystack 2

[sohanasarah]

I am having an issue running the warm-up method of the reader. I want to run the model locally. THis is the code:

retriever = InMemoryBM25Retriever(document_store = document_store)
reader = ExtractiveReader(model="./roberta-base-squad2-distilled")
reader.warm_up()

This is the error I'm getting:

Traceback (most recent call last):
  File "/mnt/c/Projects/chatbot/document_store.py", line 82, in <module>
    reader.warm_up()
  File "/home/sohanasarah/.local/lib/python3.10/site-packages/haystack/components/readers/extractive.py", line 157, in warm_up
    self.model = AutoModelForQuestionAnswering.from_pretrained(
  File "/home/sohanasarah/.local/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 566, in from_pretrained
    return model_class.from_pretrained(
  File "/home/sohanasarah/.local/lib/python3.10/site-packages/transformers/modeling_utils.py", line 2996, in from_pretrained
    raise ImportError(
ImportError: Using `low_cpu_mem_usage=True` or a `device_map` requires Accelerate: `pip install accelerate.`

How to resolve it?

[anakin87]

Probably, you can fix the issue, by installing accelerate:
pip install accelerate.

[sohanasarah]

@anakin87 I have accelerate installed. And I'm loading the reader model from my local drive. I'm assuming I'm getting the error because the model can not find the GPU. How can I set the device_map to CPU in Haystack 2?

[anakin87]

It seems the error message suggests something different (I managed to reproduce the error in Colab, without accelerate).

However, you can read about device management in Haystack 2 in this page.

[sohanasarah]

Thank you @anakin87 . I resolved the issue by setting the device to CPU.

device = ComponentDevice.from_single(Device.cpu())
reader = ExtractiveReader(model="./roberta-base-squad2-distilled", device= device)
reader.warm_up()

[satyaloka93]

I'm having the same issue with the re-ranker. I installed accelerate but then I get 'Failed to import transformers.pipelines because of the following error: No module named 'transformer_engine_extensions'. I tried all accelerate versions back to v0.2.0, where that error disappears but then I get the error 'ImportError: Using low_cpu_mem_usage=True or a device_map requires Accelerate: pip install accelerate.

There is something terrible about integrating accelerate at a version to be able to simply use the re-ranker, and wish I could set a device='cuda' without it (like I do with the embedding models).

[anakin87]

@satyaloka93 can you please provide a reproducible code example and information on how you installed haystack?

[satyaloka93]

Hi, my code works at home, but at the office I'm in a container with a mixture of system and local libraries. Everything was working until I implemented the re-ranker:

import:
from haystack.components.rankers import TransformersSimilarityRanker

instantiation:
rag_pipeline.add_component(
instance=TransformersSimilarityRanker(model="intfloat/simlm-msmarco-reranker", top_k=20), name="ranker"
)

I have seen this error with accelerate elsewhere, and people have said to uninstall transformer_engine, which I can't due to permissions (container environment is standardized). My main question is if there is a way to avoid that call to accelerate to bypass the conflict. I have used many models, embedders, etc. without having this issue.

[satyaloka93]

Ok, I solved it (after about 12 hours). I installed from source transformer-engine (https://github.com/NVIDIA/TransformerEngine), first installing flash-attn==1.0.7 (--no-build-isolation). Definitely conflict with container.