colpali v1.3 by AndrewOgn #427
Conversation
|
To check out values for tests I use code examples from here |
fastembed/late_interaction_multimodal/late_interaction_multimodal_embedding_base.py
Outdated
Show resolved
Hide resolved
fastembed/late_interaction_multimodal/late_interaction_multimodal_embedding_base.py
Outdated
Show resolved
Hide resolved
fastembed/late_interaction_multimodal/late_interaction_multimodal_embedding.py
Outdated
Show resolved
Hide resolved
| PAD_TOKEN = "<pad>" | ||
| QUERY_MARKER_TOKEN_ID = [2, 9413] | ||
| IMAGE_PLACEHOLDER_SIZE = (3, 448, 448) | ||
| EMPTY_TEXT_PLACEHOLDER = np.array([257152] * 1024 + [2, 50721, 573, 2416, 235265, 108]) |
There was a problem hiding this comment.
This is actually token ids of the following string '<image>' * 1024 + '<bos>Describe the image.\n'
Could we make it nicer? It's not really readable at the moment
EVEN_ATTENTION_MASK is also not really readable, maybe instead of having this even_attention_mask we could assign 1030 to a constant which seems to be a bit more reasonable
There was a problem hiding this comment.
That's this tokens, right. But there are some slight difference between real text input and this placeholder.
As you can see here original preprocessor does not add '\n' token there, while it should be added everywhere else. So we should refactor logic of tokenize, trigger tokenizer each time (with constant output)
descriptions docs black
it's a draft of second iteration of work on colpali #394