Skip to content

Commit

Permalink
Merge pull request milvus-io#1444 from zc277584121/master
Browse files Browse the repository at this point in the history
use image to represent colpali formula
  • Loading branch information
wxywb authored Oct 24, 2024
2 parents c3415b9 + aaf6db4 commit 4680c23
Show file tree
Hide file tree
Showing 2 changed files with 3 additions and 4 deletions.
7 changes: 3 additions & 4 deletions bootcamp/tutorials/quickstart/use_ColPali_with_milvus.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,8 @@
"\n",
"Modern retrieval models typically use a single embedding to represent text or images. ColBERT, however, is a neural model that utilizes a list of embeddings for each data instance and employs a \"MaxSim\" operation to calculate the similarity between two texts. Beyond textual data, figures, tables, and diagrams also contain rich information, which is often disregarded in text-based information retrieval.\n",
"\n",
"$$\n",
"S_{q,d} := \\sum_{i \\in |E_q|} \\max_{j \\in |E_d|} E_{q_i} \\cdot E_{d_j}^T\n",
"$$\n",
"![](../../../images/colpali_formula.png)\n",
"\n",
"MaxSim function compares a query with a document (what you're searching in) by looking at their token embeddings. For each word in the query, it picks the most similar word from the document (using cosine similarity or squared L2 distance) and sums these maximum similarities across all words in the query\n",
"\n",
"ColPali is a method that combines ColBERT's multi-vector representation with PaliGemma (a multimodal large language model) to leverage its strong understanding capabilities. This approach enables a page with both text and images to be represented using a unified multi-vector embedding. The embeddings within this multi-vector representation can capture detailed information, improving the performance of retrieval-augmented generation (RAG) for multimodal data.\n",
Expand Down Expand Up @@ -473,4 +472,4 @@
},
"nbformat": 4,
"nbformat_minor": 2
}
}
Binary file added images/colpali_formula.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit 4680c23

Please sign in to comment.