Failed to download from nim/nvidia/nv-embedqa-e5-v5:5_tokenizer

[lhrm]

I am experiencing an issue where the nemo-retriever-embedding-microservice container fails to download the required model files during startup. Here are the details:

Command:
$ docker compose -f docker-compose.yaml -f ../../local_deploy/docker-compose-nim-ms.yaml -f ../../local_deploy/docker-compose-vectordb.yaml --profile local-nim up

Error Logs:

nemo-retriever-embedding-microservice |
nemo-retriever-embedding-microservice | =========================================
nemo-retriever-embedding-microservice | == NVIDIA Retriever Text Embedding NIM ==
nemo-retriever-embedding-microservice | =========================================
nemo-retriever-embedding-microservice |
nemo-retriever-embedding-microservice | NVIDIA Release 1.0.1 (build 0cf1c7ad8e51bdffc4e4d4226735dbb7f59d70d4)
nemo-retriever-embedding-microservice | Model: nvidia/nv-embedqa-e5-v5
nemo-retriever-embedding-microservice |
nemo-retriever-embedding-microservice | Container image Copyright (c) 2016-2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
nemo-retriever-embedding-microservice | This NIM container is governed by the NVIDIA AI Product Agreement here:
nemo-retriever-embedding-microservice | https://www.nvidia.com/en-us/data-center/products/nvidia-ai-enterprise/eula/.
nemo-retriever-embedding-microservice | A copy of this license can be found under /opt/nim/LICENSE.
nemo-retriever-embedding-microservice |
nemo-retriever-embedding-microservice | The use of this model is governed by the AI Foundation Models Community License
nemo-retriever-embedding-microservice | here: https://docs.nvidia.com/ai-foundation-models-community-license.pdf.
nemo-retriever-embedding-microservice |
nemo-retriever-embedding-microservice | OTEL Logging handler requested, but Python logging auto-instrumentation not set up. Set OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true to enable logging auto-instrumentation.
nemo-retriever-embedding-microservice | downloading nim/nvidia/nv-embedqa-e5-v5:5_tokenizer
nemo-retriever-embedding-microservice | This could take a while.
nemo-retriever-embedding-microservice | --2025-01-22 23:34:03-- https://api.ngc.nvidia.com/v2/org/nim/team/nvidia/models/nv-embedqa-e5-v5/versions/5_tokenizer/zip
nemo-retriever-embedding-microservice | Resolving api.ngc.nvidia.com (api.ngc.nvidia.com)... 54.68.27.61, 44.227.231.63
nemo-retriever-embedding-microservice | Connecting to api.ngc.nvidia.com (api.ngc.nvidia.com)|54.68.27.61|:443... connected.
nemo-retriever-embedding-microservice | HTTP request sent, awaiting response... 410 Gone
nemo-retriever-embedding-microservice | 2025-01-22 23:34:04 ERROR 410: Gone.
nemo-retriever-embedding-microservice |
nemo-retriever-embedding-microservice | 2025-01-22T23:34:04Z WARNING: tools.nim.ngc_models - Failed to download from nim/nvidia/nv-embedqa-e5-v5:5_tokenizer: Failure from NGC CLI -
nemo-retriever-embedding-microservice |
nemo-retriever-embedding-microservice |
nemo-retriever-embedding-microservice exited with code 1

Environment:

• Docker Compose Version: v2.5.0

Issue:
The nemo-retriever-embedding-microservice service uses the image nvcr.io/nim/nvidia/nv-embedqa-e5-v5:1.0.1. However, during startup, it attempts to download the resource nim/nvidia/nv-embedqa-e5-v5:5_tokenizer from NVIDIA NGC. The request fails with a 410 Gone error, indicating that the resource might no longer be available.

Questions:
1.Why is the nvcr.io/nim/nvidia/nv-embedqa-e5-v5:1.0.1 image trying to download the 5_tokenizer resource?
2.Has this resource (nim/nvidia/nv-embedqa-e5-v5:5_tokenizer) been deprecated or removed from NGC?
3.Should I update the image version or modify the service configuration to resolve this issue?
4.Is there an alternative approach to pre-download the required models and avoid runtime dependency issues?
Any guidance on resolving this issue would be greatly appreciated.

Thank you!

[bryanlje]

Hi, I am facing the same issue. I am using several NIMs in my projects that are running successfully on other devices. However, when downloading these Docker images on a new device, it tries to download this 3_tokenizer_v3 file which apparently does not exist anymore. I am using the correct tag for the image.

Any help from the NVIDIA team would be much appreciated.

[dsravishankar]

I am running into same issue as well with below error message

Defaulted container "nemo-embedding-ms" out of: nemo-embedding-ms, verify-model-permissions (init)
downloading nim/nvidia/nv-embedqa-e5-v5:5_tokenizer
This could take a while.
--2025-01-27 09:28:04-- https://api.ngc.nvidia.com/v2/org/nim/team/nvidia/models/nv-embedqa-e5-v5/versions/5_tokenizer/zip
Resolving api.ngc.nvidia.com (api.ngc.nvidia.com)... 54.68.27.61, 44.227.231.63
Connecting to api.ngc.nvidia.com (api.ngc.nvidia.com)|54.68.27.61|:443... connected.
HTTP request sent, awaiting response... 410 Gone
2025-01-27 09:28:04 ERROR 410: Gone.

2025-01-27T09:28:04Z WARNING: tools.nim.ngc_models - Failed to download from nim/nvidia/nv-embedqa-e5-v5:5_tokenizer: Failure from NGC CLI -