[VLM] Merged multi-modal processor for InternVL-based models #12553
+1,395
−979
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: DarkLight1337 <[email protected]>
|
👋 Hi! Thank you for contributing to the vLLM project. Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can do one of these:
🚀 |
This was referenced Jan 29, 2025
Signed-off-by: DarkLight1337 <[email protected]>
Signed-off-by: DarkLight1337 <[email protected]>
|
|
Signed-off-by: DarkLight1337 <[email protected]>
DarkLight1337
commented
Jan 29, 2025
| @@ -723,7 +723,7 @@ See [this page](#generative-models) for more information on how to use generativ | |||
| * ✅︎ | |||
| - * `NVLM_D_Model` | |||
| * NVLM-D 1.0 | |||
| * T + I<sup>E+</sup> | |||
| * T + I<sup>+</sup> | |||
There was a problem hiding this comment.
Embedding inputs is not supported because we need image size information to calculate the number of patches for the multi-modal processor.
DarkLight1337
commented
Jan 29, 2025
Comment on lines
+260
to
+268
| @staticmethod | ||
| def flat_from_sizes(modality: str, size_per_item: torch.Tensor): | ||
| slice_idxs = [0, *accumulate(size_per_item)] | ||
| slices = [ | ||
| slice(slice_idxs[i], slice_idxs[i + 1]) | ||
| for i in range(len(size_per_item)) | ||
| ] | ||
|
|
||
| return MultiModalFieldConfig.flat(modality, slices) |
There was a problem hiding this comment.
Added factory method to avoid repeating the logic of getting the slice indices
Signed-off-by: DarkLight1337 <[email protected]>
DarkLight1337
force-pushed
the
internvl-v1
branch
from
24bf8c7 to
5132b51
Compare
Isotr0py
approved these changes
Jan 30, 2025
| min_num = hf_config.min_dynamic_patch | ||
| if dynamic_image_size is None: | ||
| dynamic_image_size = hf_config.dynamic_image_size | ||
| class InternVLProcessor: |
There was a problem hiding this comment.
I think we can also use this to replace the patch InternVLProcessor in test:
vllm/tests/models/decoder_only/vision_language/vlm_utils/model_utils.py
Lines 390 to 428 in f17f1d4
There was a problem hiding this comment.
The new implementation of HF processor flattens the pixel values, so it is incompatible with this alternative implementation.
There was a problem hiding this comment.
Actually, let me try updating the tests to not use HF processor - perhaps I can patch the generate method to be based on chat... It's difficult to keep the implementations separate as I need to copy the processing code from the README anyway.
Signed-off-by: DarkLight1337 <[email protected]>
|
Have you tested NVLM-D already? Turns out that I also have to update H2O-VL because it depends on InternVL. |
Signed-off-by: DarkLight1337 <[email protected]>
DarkLight1337
force-pushed
the
internvl-v1
branch
from
ddf55c2 to
3814c90
Compare
|
Seems that NVLM-D doesn't work yet (tested with aleiko/NVLM-D-72B-w4a16 cost about 52GB VRAM totally): |
DarkLight1337
force-pushed
the
internvl-v1
branch
3 times, most recently
from
806a9ec to
9e7aff2
Compare
Signed-off-by: DarkLight1337 <[email protected]>
Signed-off-by: DarkLight1337 <[email protected]>
DarkLight1337
force-pushed
the
internvl-v1
branch
from
9e7aff2 to
9b177b5
Compare
|
Putting it to draft until I get the tests to pass locally. |
Signed-off-by: DarkLight1337 <[email protected]>
Signed-off-by: DarkLight1337 <[email protected]>
Signed-off-by: Isotr0py <[email protected]>
|
The NVLM-D model should work on latest commit(bd67b1a) now: V0 outputs: V1 outputs: |
Signed-off-by: Isotr0py <[email protected]>
Signed-off-by: DarkLight1337 <[email protected]>
Signed-off-by: DarkLight1337 <[email protected]>
Signed-off-by: DarkLight1337 <[email protected]>
Signed-off-by: DarkLight1337 <[email protected]>
Signed-off-by: DarkLight1337 <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Update InternVL (similarly, H2OVL and NVLM-D) to use the merged multi-modal processor.
Note:
BaseProcessingInfo.get_mm_max_tokens_per_itemnow takes amm_countsargument to accommodate H2O-VL which has different processing logic depending on how many images are passed.InternVL V0 output
InternVL V1 output
@ywang96 can you help check NVLM-D? My current setup doesn't have enough memory for it.
After this PR, only Molmo and Pixtral still use the legacy input mapper in V1.