Custom LLM integration with Voice Agent

[satyam-veris]

I'm trying to integrate my custom LLM with Deepgram Voice Agent. When using their default providers (OpenAI/Anthropic), everything works fine. With OpenAI, I see this response format in the WebSocket messages:

{
    "type": "ConversationText",
    "role": "assistant", 
    "content": "How can I help you?"
}

I've set up my custom LLM REST endpoint at /chat in my localhost and exposed it publicly using NGROK. and configured the agent:

{
  "think": {
    "provider": {
      "type": "custom",
      "url": "my-server/v1/chat/completions"
    }
  }
}

My LLM receives and processes requests successfully (confirmed via logs), but the responses never appear in the WebSocket messages. What's the correct response format for custom LLM integration?

My server.js file looks like

app.post('/chat', async (req, res) => {
  try {
    const { messages, thread_id = 'default' } = req.body;
    const lastMessage = messages[messages.length - 1];
    const response = await callAgent(mongoClient, lastMessage.content, thread_id);
    console.log('response', response);
    // Format response for Deepgram
    const finalResponse = {
      id: `chatcmpl-${Date.now()}`,
      object: 'chat.completion',
      created: Math.floor(Date.now() / 1000),
      model: "custom-langgraph-agent",
      usage: {
        prompt_tokens: 0,
        completion_tokens: 0,
        total_tokens: 0
      },
      choices: [{
        message: {
          role: 'assistant',
          content: response,
          type: 'ConversationText',
        },
        finish_reason: "stop",
        index: 0
      }]
    };

    res.json(finalResponse);

  } catch (error) {
    console.error('Error:', error);
    res.status(500).json({ error: { message: 'Internal server error' } });
  }
});

Sample config for deepgram Agent:

const baseConfig = {
  type: "SettingsConfiguration",
  audio: audioConfig,
  agent: {
    listen: { model: "nova-2" },
    speak: { model: "aura-asteria-en" },
    think: {
      provider: {
        type: "custom",
        url: '<server_url>/chat',
        headers: [
          { key: 'Content-Type', value: 'application/json' }
        ],
      },
      model: "custom-langgraph-agent",
      // provider: { type: 'open_ai' },
      // model: 'gpt-4o-mini'
    },
  },
};

I am using this repository as a interface https://github.com/deepgram-devs/deepgram-voice-agent-demo.

Environment:

• Custom LLM running on Express server
• Using ngrok for testing
• Deepgram Voice Agent in browser

[deepgram-community[bot]]

Thanks for asking your question. Please be sure to reply with as much detail as possible so the community can assist you efficiently.
_{Consider joining our Discord community for more opportunity to engage with your fellow Deepgram users. You can earn points which can be redeemed for cool stuff by being active in our communities!}

[deepgram-community[bot]]

Hey there! It looks like you haven't connected your GitHub account to your Deepgram account. You can do this at https://community.deepgram.com - being verified through this process will allow our team to help you in a much more streamlined fashion.

[deepgram-community[bot]]

It looks like we're missing some important information to help debug your issue. Would you mind providing us with the following details in a reply?

• A request ID that triggered your error or issue.

[deepgram-community[bot]]

You might want to check how you are setting your configuration for the custom model.

Our Docs give an example here: https://developers.deepgram.com/docs/voice-agent-llm-models#passing-a-custom-llm-through-a-cloud-provider

Also there is a working demo of doing this with Azure Open AI here which might help you: https://github.com/deepgram-devs/voice-agent-azure-open-ai-services/blob/1ad0eee7d99cff521da0de9abaf247ad24a4e1a6/client.py#L41

Though if you are running your LLM locally, that might make this a bit more challenging as I have not tried that yet.

I've set up my custom LLM REST endpoint at /chat in my localhost and exposed it publicly using NGROK. and configured the agent:
There might be something on the NGROK side you need to look at to enable the connection.

This message was sent by John Vajda from Deepgram, via our community automation.