[Bug]: Custom Endpoint LocalAI "Failed to fetch models from OpenAI API Request failed with status code 404"

[K-J-VV]

What happened?

Having a hard time figuring out how to use LocalAI as a custom endpoint with LibreChat

Keep getting an error stating that the models were not able to be fetched, but the model does show in LibreChat and is selectable. Whenever I try to type anything to the chat, the GPT just ends and then this error is displayed in the log

example of .env:

PROXY=
OPENAI_FORCE_PROMPT=true
OPENAI_REVERSE_PROXY=https://localai.example.com/v1/chat/completions   # I have tried /v1 as well
OPENAI_MODELS=gpt-3.5-turbo
PLUGIN_MODELS=gpt-3.5-turbo
DEBUG_PLUGINS=true
GALLERIES=[{"name":"model-gallery", "url":"github:go-skynet/model-gallery/index.yaml"}]
PRELOAD_MODELS=[{"url": "github:go-skynet/model-gallery/gpt4all-j.yaml", "name": "gpt-3.5-turbo"}]

example of librechat.yaml :

version: 1.0.1
cache: true
endpoints:
  custom:
     # gpt4all-j
    - name: "GPT"
      apiKey: "xxxxx"
      baseURL: "https://localai.example.com/v1/chat/completions"
      models:
        default: ["gpt-3.5-turbo"]
        fetch: true
      titleConvo: true
      titleModel: "gpt-3.5-turbo"
      summarize: true
      summaryModel: "gpt-3.5-turbo"
      forcePrompt: true
      modelDisplayLabel: "GPT"

models at https://localai.example.com/v1/models :

{
    "object": "list",
    "data": [
        {
            "id": "gpt-3.5-turbo",
            "object": "model"
        }
    ]
}

Steps to Reproduce

1. Create docker containers for LocalAI and LibreChat
2. Use librechat.yaml to point to custom endpoint of LocalAI models

What browsers are you seeing the problem on?

No response

Relevant log output

error: Failed to fetch models from OpenAI API Request failed with status code 404
2024-01-25 02:00:46 error: [MeiliMongooseModel.findOneAndUpdate] Convo not found in MeiliSearch and will index 98932e9c-7f7c-4a08-b087-fbed41f18046 Document `98932e9c-7f7c-4a08-b087-fbed41f18046` not found.

Screenshots

Code of Conduct

• I agree to follow this project's Code of Conduct

[danny-avila]

The model fetch fail: it is literally calling OpenAI and that's why it fails, it may be unrelated to your issue.

Check the logs in /api/logs both debug and error logs (app/api/logs within container of LibreChat if using docker), lmk if you find anything.

Do you get no output at all? An immediate empty response? Or does it just hang with the blinking cursor?

I have not tested LocalAI with LibreChat in a long time but I remember it working. I will try to get it setup again to test for you.

What do the LocalAI logs look like, does it ever reach the server?

If LocalAI is running on docker, try http://host.docker.internal:PORT/v1/chat/completions

[K-J-VV]

"Do you get no output at all? An immediate empty response? Or does it just hang with the blinking cursor?"

After I submit chat, I just get constant blinking and no response

"What do the LocalAI logs look like, does it ever reach the server?"

The logs for LocalAI show no errors, or anything. So it doesn't look as if anything is happening on the LocalAi end of things

Here is the debug from LibreChat:

cat debug-2024-01-25.log
2024-01-25 03:20:51 info: [Optional] Redis not initialized. Note: Redis support is experimental.
2024-01-25 03:20:52 info: Connected to MongoDB
2024-01-25 03:20:52 debug: [indexSync] There are 4 messages in the database, 4 indexed
2024-01-25 03:20:52 debug: [indexSync] There are 2 convos in the database, 2 indexed
2024-01-25 03:20:52 info: Loaded custom config file
2024-01-25 03:20:52 info: Server listening on all interfaces at port 3080. Use http://localhost:3080 to access it
2024-01-25 03:20:53 debug: [validateJson] files
["\"Ai_PDF.json\"","\"BrowserOp.json\"","\"Dr_Thoths_Tarot.json\"","\"DreamInterpreter.json\"","\"VoxScript.json\"","\"askyourpdf.json\"","\"drink_maestro.json\"","\"earthImagesAndVisualizations.json\"","\"image_prompt_enhancer.json\"","\"qrCodes.json\"","\"scholarai.json\"","\"uberchord.json\"","\"web_search.json\""]
2024-01-25 03:20:54 error: Request failed with status code 404
2024-01-25 03:20:54 debug: [validateJson] files
["\"Ai_PDF.json\"","\"BrowserOp.json\"","\"Dr_Thoths_Tarot.json\"","\"DreamInterpreter.json\"","\"VoxScript.json\"","\"askyourpdf.json\"","\"drink_maestro.json\"","\"earthImagesAndVisualizations.json\"","\"image_prompt_enhancer.json\"","\"qrCodes.json\"","\"scholarai.json\"","\"uberchord.json\"","\"web_search.json\""]
2024-01-25 03:21:11 debug: [AskController]
{
  text: "hello",
  conversationId: null,
  endpoint: "GPT",
  endpointType: "custom",
  chatGptLabel: null,
  promptPrefix: null,
  resendImages: false,
  imageDetail: "auto",
    modelOptions.model: "gpt-3.5-turbo",
    modelOptions.temperature: 1,
    modelOptions.top_p: 1,
    modelOptions.presence_penalty: 0,
    modelOptions.frequency_penalty: 0,
}
2024-01-25 03:21:12 debug: [BaseClient] Loading history:
{
  conversationId: "be00f302-d934-45a9-a5df-d37e59c26bd2",
  parentMessageId: "00000000-0000-0000-0000-000000000000",
}

I think the issue must be something being wrong with the content of the custom config file (librechat.yaml) that links LibreChat to the custom endpoint... Or something in the .env file. Considering there's no issues with Mistral and other endpoints, it all should work with LocalAI and its just a matter of setting a few variables correctly.

I did see it looks like this problem was resolved in an older issue: #1027

[chnyda]

Did you get it to work @K-J-VV ?

[badcom]

Do you want to fetch the models? Otherwise, then try changing the fetch key to false

[danny-avila]

For the record, I didn't notice before but you shouldn't use the env variables in combination with the custom endpoints as they can introduce conflicts

OPENAI_FORCE_PROMPT=true
OPENAI_REVERSE_PROXY=https://localai.example.com/v1/chat/completions # I have tried /v1 as well

[chnyda]

thanks. It is now working.