[Model] Extend Ultravox to accept audio longer than 30s

[farzadab]

Currently the Ultravox model input is capped to 30 seconds and extra audio is truncated (AFAIK). Also each sample is fed to Whisper individually (without being batched).

This PR allows using longer audio by chunking them first, using Whisper encoder in batch mode, and then concatenates them.

TODO:

• processors on HF still need to be updated in tandem with this PR.
• run evaluations with the updated model to verify the changes.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[mgoin]

FYI @NickLucche for the usage of whisper

[NickLucche]

Thanks for the contrib!
What is the chunking logic for tiling the audio? Feel free to link the hf processor PR.

[farzadab]

re @NickLucche: Here's the processor link: https://huggingface.co/fixie-ai/ultravox-v0_3-llama-3_2-1b/blob/main/ultravox_processing.py#L209

The logic: for each audio, split to 30 second chunks (but do not pad the last item to 30s, which is the same as before).
Then we flatten and batch everything up and run Whisper as if they were separate audios. We use audio_lens to compute an attention_mask for the last chunk per audio. The final embeddings are then concatenated.

There are other ways we could've done this, but it matches what we do on the Ultravox side for both some fine-tuning that we do and evals. If we end up updating those I'll update VLLM as well.

Also, note that since we don't pad the last chunk, and since in most cases we have smaller than 30s audio, the number of frames do not match across samples. I didn't see a collator anywhere that I could update. I'm suspecting that I'll have to update _process_audio_input further to handle that. Updated _process_audio_input.

[NickLucche]

Ok I see then that's a naive chunking where you don't account for splitting mid-word nor you have any overlap and/or prompt from previous chunk.

This case seems much easier to handle vllm-side, given changes are already in hf. Let's just make sure the batched whisper forward is accounted for by the initial profiler run to avoid oom.

[ywang96]

Hello! @farzadab I left a few comments regarding the CI failure. Thanks for your contribution!

[farzadab]

The tests are finally passing, yay!
The PR is ready to merge.

[DarkLight1337]

Nice, let's merge this then.

[varun-sundar-rabindranath]

Hey guys, looks like this PR broke the Ultravox LoRA tests. Both V0 and V1 tests/lora/test_ultravox.py "were" failing.
I can't seem to repro the V0 test failure locally.
I can repro the V1 test failure - The test hits the assert

vllm/vllm/v1/worker/gpu_model_runner.py

Line 1380 in 53be4a8

assert len(dummy_encoder_outputs) == max_num_mm_items, (

. If I remove the assert the test works fine. @ywang96 @DarkLight1337 can you please take a look when you get a chance. Thanks!
cc @jeejeelee @robertgshaw2-redhat