User Audio Input transcription issue

[kamjony]

System Info

Dart

Related Components

• doc-loaders
• doc-transformers
• prompts
• llms
• chat-models
• output-parsers
• chains
• memory
• stores
• embeddings
• retrievers
• agents

Reproduction

await client.connect();
await client.updateSession(
    instructions: gptInstructions,
    modalities: [Modality.audio],
    voice: Voice.shimmer,
    inputAudioFormat: AudioFormat.pcm16,
    outputAudioFormat: AudioFormat.pcm16,
    inputAudioTranscription: InputAudioTranscriptionConfig(
      // enabled: true,
      model: 'whisper-1',
    )
);

Expected behavior

The event "type:"conversation.item.input_audio_transcription.completed", is never fired. But if I Use chatgpt playground, I can see this event has been fired. I need audio transcription from user audio. How to achieve it?

[davidmigloz]

There seems to be an issue with the input_audio_transcription param:

{type: error, event_id: event_AgoCpMJ7LkKOPOCfcNLn9, error: {type: invalid_request_error, code: unknown_parameter, message: Unknown parameter: 'session.input_audio_transcription.enabled'., param: session.input_audio_transcription.enabled, event_id: evt_nAh8N2BDAnkzVHMeH}}

I'll look further into it.

[davidmigloz]

I'll remove the enable parameter which is not required anymore. Apart from that, the request looks to be according to the spec.

It seems more people are facing this issue:
https://community.openai.com/t/realtime-api-session-update-doesnt-change-input-audio-format/967077

Most of the issues when not getting a transcription have to do with the input audio that is being passed.
Write it to a file and listen to it, maybe you’ll spot some errors.
Make sure that the samplerate is 24000 Hz as the API requires this.
Make sure that the audio doesn’t sound distorted, cut out, speed up or down or pitched up/down.

Let me know if you manage to solve it.

[kamjony]

@davidmigloz I did everything that you suggested above but was unable to get "type:"conversation.item.input_audio_transcription.completed".
So, I just created my own client and consumed the realtime api just to see if it was actually a problem with the package. After several hours of debugging, I found out that, if you send full audio in conversation.create, transcription is not generated by openAI. But, if you append the audio PCM using input_audio_buffer.append and then commit the audio using "type": "input_audio_buffer.commit", then the transcription is generated. I will try with the package client soon and report back.

[davidmigloz]

Interesting, thanks for sharing! Let me know the results when you try it with openai_realtime_dart client

[dominikmucklow]

Has anyone been able to resolve this issue? We are receiving the audio just fine but cannot locate a transcription in any of the event logs.

[kamjony]

@davidmigloz @dominikmucklow With this package, I was unable to get transcription as there are no separate function to commit the audio before generating a gpt response. So, I have manually implemented chatgpt realtime following their docs. Now, I get transcription and have a freedom to generate response when I want to.

[davidmigloz]

Thanks for the feedback @kamjony! I'll review their latest spec, maybe they added that method after my initial implementation.

[dominikmucklow]

@kamjony Thanks for the info, and glad to know that it can work. Manual implementation is over my head, so I'll keep following updates to this package. @davidmigloz keep up the good work! Here's a lunch on me.

[davidmigloz]

Thanks! I'll try to look into it this weekend