[Question]: Title Generation - Custom endpoint

[QuartzWarrior]

What is your question?

Why am I getting this error, and how can I fix it?

I'm assuming there must be something wrong with the endpoint used, but not entirely sure.

I'm trying to get title generation to work on my custom endpoint ( the only endpoint enabled in my configs ) but always get the following errors, even though I don't have OpenAI enabled.

More Details

received error

2024-12-10 03:25:24 warn: [OpenAIClient.chatCompletion][create] API error
2024-12-10 03:25:24 error: [OpenAIClient.chatCompletion] Unhandled error type Error: 422 status code (no body)
2024-12-10 03:25:24 error: [OpenAIClient] There was an issue generating the title with the completion method Error: 422 status code (no body)

librechat.yaml looks similar to this:

 custom:
    - name: 'AI'
      apiKey: 'user_provided'
      baseURL: 'http://localhost:90/v1/chat/completions'
      models: []
        fetch: false
      titleConvo: true
      titleModel: 'ministral-3b-latest'
      titleMethod: "completion"
      directEndpoint: true
      modelDisplayLabel: ' AI'

yes, I did provide the api key.

My backend is running on localhost:80, frontend running on localhost:90

---

My api only allows requests that have the Authorization header with the value being Bearer sk-*****

What is the main subject of your question?

UI, Endpoints

Screenshots

Code of Conduct

• I agree to follow this project's Code of Conduct

[danny-avila]

It's using OpenAI API spec, which is why you see that in the logs. It's having issue with the request made for titling. This is what the payload looks like (for me using groq):

2024-12-10T13:18:32.162Z debug: [OpenAIClient] chatCompletion
{
  baseURL: "https://api.groq.com/openai/v1",
    modelOptions.model: "mixtral-8x7b-32768",
    modelOptions.user: "userID",
    modelOptions.temperature: 0.2,
    modelOptions.presence_penalty: 0,
    modelOptions.frequency_penalty: 0,
    modelOptions.max_tokens: 16,
    // 1 message(s)
    modelOptions.messages: [{"role":"system","content":"Please generate a concise, 5-word-or-less title for the conversation, us... [truncated]],
}

It's possible your endpoint doesn't support one of those parameters.

[QuartzWarrior]

just tried it with those params, and seems to work perfectly fine.

fyi, ministral-3b-latest works aswell

[danny-avila]

can you share your debug logs when this happens?

If using docker, it's ./logs

if using npm directly, it's ./api/logs

[QuartzWarrior]

2024-12-10T16:26:40.202Z debug: [OpenAIClient] chatCompletion
{
  baseURL: "http://localhost:90/v1",
    modelOptions.model: "ministral-3b-latest",
    modelOptions.user: "669af6f080b0461a7c31ec44",
    modelOptions.temperature: 0.2,
    modelOptions.presence_penalty: 0,
    modelOptions.frequency_penalty: 0,
    modelOptions.max_tokens: 16,
    // 1 message(s)
    modelOptions.messages: [{"role":"system","content":"Please generate a concise, 5-word-or-less title for the conversation, us... [truncated]],
}
2024-12-10T16:26:40.202Z debug: Making request to http://localhost:90/v1/chat/completions
2024-12-10T16:26:40.225Z warn: [OpenAIClient.chatCompletion][create] API error
2024-12-10T16:26:40.225Z error: [OpenAIClient.chatCompletion] Unhandled error type Error: 422 status code (no body)
2024-12-10T16:26:40.225Z error: [OpenAIClient] There was an issue generating the title with the completion method Error: 422 status code (no body)
2024-12-10T16:26:40.225Z debug: [OpenAIClient] Convo Title: New Chat
2024-12-10T16:26:40.225Z debug: [saveConvo] api/server/services/Endpoints/openAI/

[danny-avila]

I suspected this, it's making a request to the wrong endpoint:
Making request to http://localhost:90/v1/chat/completions

it seems the directEndpoint option is not taking effect when creating titles, this will need a fix

      baseURL: 'http://localhost:800/chat/completions'
# ...
      directEndpoint: true

[QuartzWarrior]

I suspected this, it's making a request to the wrong endpoint: Making request to http://localhost:90/v1/chat/completions

it seems the directEndpoint option is not taking effect when creating titles, this will need a fix

      baseURL: 'http://localhost:800/chat/completions'
# ...
      directEndpoint: true

im sorry, forgot to mention, I changed the port to 90 and added /v1, its making the request to the correct place.

here's the full debug with the first request:

2024-12-10T18:23:56.857Z debug: [AskController]
{
  text: "reddit",
  conversationId: null,
  endpoint: "Shard AI",
  endpointType: "custom",
  resendFiles: true,
    modelOptions.model: "ministral-8b-latest",
  modelsConfig: "exists",
}
2024-12-10T18:23:56.859Z debug: [BaseClient] Loading history:
{
  conversationId: "4fed15c3-13d8-436b-863e-2b3a568ce8fe",
  parentMessageId: "00000000-0000-0000-0000-000000000000",
}
2024-12-10T18:23:56.861Z debug: [BaseClient] Context Count (1/2)
{
  remainingContextTokens: 4087,
  maxContextTokens: 4095,
}
2024-12-10T18:23:56.861Z debug: [BaseClient] Context Count (2/2)
{
  remainingContextTokens: 4087,
  maxContextTokens: 4095,
}
2024-12-10T18:23:56.861Z debug: [BaseClient] tokenCountMap:
{
  05a689d2-b328-4247-a995-e350a71231e9: 5,
}
2024-12-10T18:23:56.861Z debug: [BaseClient]
{
  promptTokens: 8,
  remainingContextTokens: 4087,
  payloadSize: 1,
  maxContextTokens: 4095,
}
2024-12-10T18:23:56.861Z debug: [BaseClient] tokenCountMap
{
  05a689d2-b328-4247-a995-e350a71231e9: 5,
  instructions: undefined,
}
2024-12-10T18:23:56.861Z debug: [BaseClient] userMessage
{
  messageId: "05a689d2-b328-4247-a995-e350a71231e9",
  parentMessageId: "00000000-0000-0000-0000-000000000000",
  conversationId: "4fed15c3-13d8-436b-863e-2b3a568ce8fe",
  sender: "User",
  text: "reddit",
  isCreatedByUser: true,
  tokenCount: 5,
}
2024-12-10T18:23:56.861Z debug: [OpenAIClient] chatCompletion
{
  baseURL: "http://localhost:90/v1",
    modelOptions.model: "ministral-8b-latest",
    modelOptions.user: "669af6f080b0461a7c31ec44",
    modelOptions.stream: true,
    // 1 message(s)
    modelOptions.messages: [{"role":"user","content":"reddit"}],
}
2024-12-10T18:23:56.862Z debug: Making request to http://localhost:90/v1/chat/completions
2024-12-10T18:23:56.864Z debug: [saveConvo] api/app/clients/BaseClient.js - saveMessageToDatabase #saveConvo
2024-12-10T18:24:06.876Z debug: [OpenAIClient] chatCompletion response
{
  object: "chat.completion",
  service_tier: null,
    usage.completion_tokens: 512,
    usage.prompt_tokens: 5,
    usage.total_tokens: 517,
  id: "chatcmpl-kUdSFAC039ZX6uO750bcLTsdJ1aQ",
  // 1 choice(s)
  choices: [{"message":{"role":"assistant","content":"Reddit is a popular social news aggregation, web content r... [truncated]],
  created: 1733855046,
  model: "ministral-8b-latest",
  system_fingerprint: "fp_gXq8JFnqCE",
}
2024-12-10T18:24:06.876Z debug: [spendTokens] conversationId: 4fed15c3-13d8-436b-863e-2b3a568ce8fe | Context: message | Token usage:
{
  promptTokens: 8,
  completionTokens: 512,
}
2024-12-10T18:24:06.879Z debug: [spendTokens] No transactions incurred against balance
2024-12-10T18:24:06.882Z debug: [saveConvo] api/app/clients/BaseClient.js - saveMessageToDatabase #saveConvo
2024-12-10T18:24:06.886Z debug: [AskController] Request closed
2024-12-10T18:24:06.888Z debug: [OpenAIClient] chatCompletion
{
  baseURL: "http://localhost:90/v1",
    modelOptions.model: "ministral-3b-latest",
    modelOptions.user: "669af6f080b0461a7c31ec44",
    modelOptions.temperature: 0.2,
    modelOptions.presence_penalty: 0,
    modelOptions.frequency_penalty: 0,
    modelOptions.max_tokens: 16,
    // 1 message(s)
    modelOptions.messages: [{"role":"system","content":"Please generate a concise, 5-word-or-less title for the conversation, us... [truncated]],
}
2024-12-10T18:24:06.888Z debug: Making request to http://localhost:90/v1/chat/completions
2024-12-10T18:24:06.891Z warn: [OpenAIClient.chatCompletion][create] API error
2024-12-10T18:24:06.891Z error: [OpenAIClient.chatCompletion] Unhandled error type Error: 422 status code (no body)
2024-12-10T18:24:06.892Z error: [OpenAIClient] There was an issue generating the title with the completion method Error: 422 status code (no body)
2024-12-10T18:24:06.892Z debug: [OpenAIClient] Convo Title: New Chat
2024-12-10T18:24:06.892Z debug: [saveConvo] api/server/services/Endpoints/openAI/addTitle.js

[danny-avila]

can you inspect the error being displayed? only way to determine at this point is to inspect the error with breakpoints on LC side, or if the other server is on localhost, any idea why it's rejecting it with 422 on its end?

I will try to add better logging for those errors