Semantic Search LLM and Vector Databases

[wassfila]

Introduction

• Semantic search can be referred to as the extension to include similarities and meanings in the search match as opposed to Full text search which only finds a characters match.
• Semantic search uses word tokens for fuzzy and synonyms search which make it use the same infrastructure as LLMs for words tokens management using vectors databases or engines
• Preference of Semantic search over LLM chat query is a matter of result representation and use case.
• Vector databases are the required underlying framework for both instant search and LLMs, they offer eased integrations e.g. with OpenAI, HuggingFace,...
• Vector databases support a so called Generative Search which pipes search results through LLM models

Features

It is still possible to distinguish features :

• Common
  • Typo tolerance
  • Synonyms
  • Multitenancy
• Instant search
  • Search as you type
  • Highlighting
  • Filtering, Sorting
  • Faceting : clustering results in groups
  • Structured Data Admin interface
• Chat engine
  • unified meaningful answer to a question
  • Natural language for multi conditional search query
  • Unstructured Data

Instant Search

• Meilisearch https://www.meilisearch.com/docs/learn/what_is_meilisearch/overview
• Typesense https://typesense.org/docs/
• Solr https://solr.apache.org/features.html
• Algoia https://www.algolia.com/doc/
• Elastic Search https://www.elastic.co/subscriptions

open source popularity

• meilisearch 36k https://github.com/meilisearch/meilisearch
• typesense 13k https://github.com/typesense/typesense
• solr 700 https://github.com/apache/solr

comparisions

• https://www.meilisearch.com/docs/learn/what_is_meilisearch/comparison_to_alternatives#comparison-table

LLM and ML engines

• Langchain https://docs.langchain.com/docs/
• Hugging face https://huggingface.co/
• https://openai.com/product

Langchain

Lang chain is a framework abstracting other libraries to speed up development of LLM Apps

Hugging face

Machine Learning Models framework for building, training and deployment

Vector Databases

• Milvus https://milvus.io/
• Weaviate https://weaviate.io/developers/weaviate
• Pinecone https://docs.pinecone.io/docs/overview

open source popularity

• Milvus 18k https://github.com/milvus-io/milvus
• Weaviate 5k https://github.com/weaviate/weaviate
• faiss (facebook) 21k https://github.com/facebookresearch/faiss

vectorization

• Images to vectors

  • CNN models : VGG, ResNet, Inception
  • OpenCV : SIFT, SURF, ORB
  • FastText
  • ImageNet
  • Faiss
  • Hugging face transformers (pre-trained models) : ViT, DeiT

• Text

  • Word Embeddings : Word2Vec, GloVe, FastText
  • BERT

references

• https://dev.to/josethz00/vector-databases-5df1

[wassfila]

• https://github.com/MicroWebStacks/meilisearch-test
• https://github.com/MicroWebStacks/milvus-test

[wassfila]

transferred to https://github.com/orgs/VectorMind/discussions/1