Huggingface-TGI: Local model not recognized

[n-morgan]

Hi all, I am trying to use huggingface TGI with a local model linked here https://huggingface.co/teknium/OpenHermes-2.5-Mistral-7B., however, after running the following script:

model=/data/home/[REDACTED]/Development/HuggingFace-TGI/OpenHermes

volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run

docker run -e HF_HUB_OFFLINE=1 --gpus all --shm-size 1g -p 5050:80 -v $volume:/data
ghcr.io/huggingface/text-generation-inference:2.2.0
--model-id $model

I get the following error:

huggingface_hub.errors.HFValidationError: Repo id must be in the form 'repo_name' or 'namespace/repo_name': '/data/home/[REDACTED]/Development/HuggingFace-TGI/OpenHermes'. Use repo_type argument if needed.

Error: DownloadError

Does anyone know why my model is not being recognized? My end goal is to build a docker container which can serve this model upon spinup to the port indicated by the container.

[fawern]

When using Hugging Face's Text Generation Inference with a local model, it's important to ensure that the model directory is correctly specified and accessible within the Docker container.

[fawern]

Ensure your model files ( config.json, pytorch_model.bin, tokenizer.json) are in a directory, say /path/to/your/model.
Ru the docker:
docker run --gpus all --shm-size 1g -p 5050:80 -v /path/to/your/model:/data/model ghcr.io/huggingface/text-generation-inference:2.2.0 --model-id /data/model
Ensure the model directory contains all necessary files and that the --model-id path matches the mounted directory inside the container.