Skip to content

Commit

Permalink
support LMStudio when using ChatOpenAI (#243)
Browse files Browse the repository at this point in the history
  • Loading branch information
brainlid authored Jan 30, 2025
1 parent f9781d3 commit abf059e
Show file tree
Hide file tree
Showing 2 changed files with 9 additions and 4 deletions.
5 changes: 3 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ Currently supported AI services:
- Ollama
- Mistral
- Bumblebee self-hosted models - including Llama, Mistral and Zephyr
- [LMStudio](https://lmstudio.ai/docs/api/endpoints/openai) via their OpenAI compatibility API

**LangChain** is short for Language Chain. An LLM, or Large Language Model, is the "Language" part. This library makes it easier for Elixir applications to "chain" or connect different processes, integrations, libraries, services, or functionality together with an LLM.

Expand Down Expand Up @@ -124,11 +125,11 @@ A list of models to use:

## Prompt caching

ChatGPT and Claude both offer prefix-based prompt caching, which can offer cost and performance benefits for longer prompts. Gemini offers context caching, which is similar.
ChatGPT and Claude both offer prefix-based prompt caching, which can offer cost and performance benefits for longer prompts. Gemini offers context caching, which is similar.

- [ChatGPT's prompt caching](https://openai.com/index/api-prompt-caching/) is automatic for prompts longer than 1024 tokens, caching the longest common prefix.
- [Claude's prompt caching](https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching) is not automatic. It's prefixing processes tools, system, and then messages, in that order, up to and including the block designated with {"cache_control": {"type": "ephemeral"}} . See LangChain.ChatModels.ChatAnthropicTest and for an example.
- [Gemini's context caching]((https://ai.google.dev/gemini-api/docs/caching?lang=python)) requires a seperate call which is not supported by Langchain.
- [Gemini's context caching]((https://ai.google.dev/gemini-api/docs/caching?lang=python)) requires a seperate call which is not supported by Langchain.

## Usage

Expand Down
8 changes: 6 additions & 2 deletions lib/chat_models/chat_open_ai.ex
Original file line number Diff line number Diff line change
Expand Up @@ -312,9 +312,9 @@ defmodule LangChain.ChatModels.ChatOpenAI do
end
end)
|> Enum.reverse(),
response_format: set_response_format(openai),
user: openai.user
}
|> Utils.conditionally_add_to_map(:response_format, set_response_format(openai))
|> Utils.conditionally_add_to_map(
:reasoning_effort,
if(openai.reasoning_mode, do: openai.reasoning_effort, else: nil)
Expand Down Expand Up @@ -357,7 +357,11 @@ defmodule LangChain.ChatModels.ChatOpenAI do
end

defp set_response_format(%ChatOpenAI{json_response: false}) do
%{"type" => "text"}
# NOTE: The default handling when unspecified is `%{"type" => "text"}`
#
# For improved compatibility with other APIs like LMStudio, this returns a
# `nil` which has the same effect.
nil
end

defp get_tool_choice(%ChatOpenAI{
Expand Down

0 comments on commit abf059e

Please sign in to comment.