[Bugfix] handle alignment of arguments in convert_sparse_cross_attention_mask_to_dense

[tjohnson31415]

Reproducing the bug reqiures a batch with a text-only request and a request with an image. With the OpenAI server, this can happen when under load, but it is easier to reproduce in offline mode:

from vllm import LLM, SamplingParams
from vllm.multimodal.utils import fetch_image

image_url = "https://upload.wikimedia.org/wikipedia/commons/d/da/2015_Kaczka_krzy%C5%BCowka_w_wodzie_%28samiec%29.jpg"
image_data = fetch_image(image_url)

model_name = "meta-llama/Llama-3.2-11B-Vision-Instruct"

llm = LLM(
    model=model_name,
    max_model_len=4096,
    max_num_seqs=2,
    enforce_eager=True,
)

sampling_params = SamplingParams(
    temperature=0.0,
    max_tokens=128
)

outputs = llm.generate(
    prompts=[
        {
            "prompt": "What is the capital of Spain?",
        },
        {
            "prompt": "Analyze this image <|image|>. What do you see?",
            "multi_modal_data": {
                "image": image_data,
            },
        },
    ],
    sampling_params=sampling_params
)

This would have been a crash before #11939 but now results in an AssertionError. The reason is that the num_tiles: List[List[int]] passed in to convert_sparse_cross_attention_mask_to_dense would not have any entry in the list for a sequence that is text only. The inputs to the function would be like:

sparse_mask = [[], [[5, -1]]]
num_tiles = [[4]]
lengths = [7, 12]

The fix in this PR is to skip [] entries in sparse_mask for text-only requests.

A better fix may be to have num_tiles created with an entry for each sequence, but I didn't see where to make that change.

Potential fix for #10648

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

• Add ready label to the PR
• Enable auto-merge.

🚀

[DarkLight1337]

Thanks for fixing!

[heheda12345]

LGTM! Thank you for the bug fix.
@DarkLight1337 Does @large_gpu_test(min_gb=48) means this new test will be skipped during CI? If that is true, I think we should implement some test for get_cross_attention_mask (and also get_cross_attention_states & get_full_text_row_masked_out_mask if possible)

[DarkLight1337]

@DarkLight1337 Does @large_gpu_test(min_gb=48) means this new test will be skipped during CI?

Yes, that is correct.

[heheda12345]

@tjohnson31415 Can you change e2e test to a test for get_cross_attention_mask? (and also get_cross_attention_states & get_full_text_row_masked_out_mask if possible)

[wallashss]

Hey @heheda12345 , I added some tests that would be enough to prevent regression on this issue. Please, see if you agree.

Thanks!

cc @tjohnson31415

[heheda12345]

LGTM! Thanks for the bug fix and new tests!