Running Ollama Locally

[mattv8]

Hey all,
I'm exploring the possibility of connecting LibreChat running on a remote server to Ollama on a local PC. Specifically, I want to run LibreChat on a data center server without GPU capabilities and have it send requests to Ollama on my local machine, which has an RTX 4090 Ti(for example). I know this setup is possible with LobeChat (I've got it tested and working), but I'd prefer to implement it with LibreChat. Is this currently supported in LibreChat, or would it require additional development?

[fuegovic]

It should be possible by adding it as a custom endpoint in the librechat.yaml file and specifying its baseURL

see:
https://www.librechat.ai/docs/configuration/librechat_yaml/setup
https://www.librechat.ai/docs/configuration/librechat_yaml/ai_endpoints/ollama

[mattv8]

So I've played around with several versions of my librechat.yaml file baseURL part, like below, but none of them seem to work. What might I be missing?

apiKey: "ollama"
      baseURL: "http://127.0.0.1:11434/v1/chat/completions"
      baseURL: "http://host.docker.internal:11434/v1/chat/completions"
      baseURL: "http://localhost:11434/v1/"

All three show the error:

Something went wrong. Here's the specific error message we encountered: An error occurred while processing your request. Please contact the Admin.

At the very least, is there a way to make the error output more verbose?

[danny-avila]

If you're running Ollama separately from LibreChat (not coupled in the same docker compose config), then you need to expose the Ollama API Server

this can be done setting an environment variable for the terminal before running ollama

On windows powershell, I do this like so:

PS C:\Users\danny> $env:OLLAMA_HOST = "0.0.0.0:11434"

and then I run

ollama serve

More info: https://github.com/ollama/ollama/blob/main/docs/faq.md#how-do-i-configure-ollama-server

[Medenor]

Hello,

I am experiencing difficulties accessing locally-installed Ollama or Mistral with LibreChat. I am using MacOS, and Ollama software (including Ollama 3.2 and Mistral 7B) is running separately from LibreChat, which is being operated through Docker.

I exposed the Ollama API server using:

launchctl setenv OLLAMA_HOST "0.0.0.0:11434"

I added Ollama as a custom endpoint in the librechat.yaml:

- name: "Ollama"
      apiKey: "ollama"
      # use 'host.docker.internal' instead of localhost if running LibreChat in a docker container
      baseURL: "http://localhost:11434/v1/chat/completions" 
      models:
        default: [
          "llama3:latest",
          "mistral"
          ]
        # fetching list of models is supported but the `name` field must start
        # with `ollama` (case-insensitive), as it does in this example.
        fetch: true
      titleConvo: true
      titleModel: "current_model"
      summarize: false
      summaryModel: "current_model"
      forcePrompt: false
      modelDisplayLabel: "Ollama"

After restarting Ollama and Docker, I am able to access Ollama on MacOS through my phone using Mac IP:11434 ("Ollama is running"), indicating it is exposed. I'm also able to access both models with Open WebUI and Obsidian. However, it does not function in LibreChat. It appears in menu but when I send any prompt, I receive an error message:

Something went wrong. Here's the specific error message we encountered: An error occurred while processing your request. Please contact the Admin.

Do you have any idea why it does not function?

[exterity]

I’m running Windows and have LibreChat set up to run in a Docker container.
Just use this: # use 'host.docker.internal' instead of localhost if running LibreChat in a docker container
baseURL: "http://host.docker.internal:11434/v1/chat/completions" as mentioned and it works just fine then.

[Medenor]

I apologize for missing that detail! I'm glad to see it's working now.