TabbyML support for Azure OpenAI API models?

[mornante-basson-1nebula]

Hi

I am trying to connect a Model using the Model HTTP API guides, I have OpenAI models in Azure that I would like to connect with TabbyML.

So far I have configured the config.toml file with the following secrets (can see below) as per guide (I have also tried removing the suffixes as described in the guides) but still no luck,

# Chat model
[model.chat.http]
kind = "openai/chat"
model_name = "gpt-4o"
api_endpoint = "https://<my-resource-name>.cognitiveservices.azure.com/openai/deployments/gpt-4o?api-version=2024-08-01-preview"
api_key = "<my-secret>"

# Completion model
[model.completion.http]
kind = "openai/completion"
model_name = "gpt-4o"
api_endpoint = "https://<my-resource-name>.cognitiveservices.azure.com/openai/deployments/gpt-4o/chat?api-version=2024-08-01-preview"
api_key = "<my-secret>"

# Embedding model
[model.embedding.http]
kind = "openai/embedding"
model_name = "text-embedding-3-large"
api_endpoint = "https://<my-resource-name>.cognitiveservices.azure.com/openai/deployments/text-embedding-3-large?api-version=2023-05-15"
api_key = "<my-secret>"

I assume this is because the API's provided by Azure with the "api-version" query parameters is what is causing the issue?

TabbyML starts up correctly when I first booted the server with the following command:

docker run -it -p 8080:8080 -v %USERPROFILE%/.tabby:/data registry.tabbyml.com/tabbyml/tabby serve

But then I get the following error for all my models under they Information -> System page (/system):

Error 404: {"error":{"code":"404","message": "Resource not found"}}

Any help would be appreciated (if its even possible using the models from Azure)

[wsxiaoys]

Hi - using azure openai service is currently not supported.

Filing #3668 to track the FR, cc @zwpaper as FYI