blog: Why HNSW is not the answer

[gaocegege]

HNSW has become the de facto standard for many vector databases. It is often considered:

particularly well-suited for large-scale vector similarity searches due to its multi-layered graph structure, which efficiently navigates vector embeddings, and its support for the incremental addition of new data points.

However, we challenge this assumption. While HNSW performs well on smaller datasets, it struggles to scale effectively when dataset sizes grow significantly. This is largely due to its reliance on memory-based indexing, which becomes impractical for large-scale applications compared to disk-based solutions.

This contrast could serve as the foundation for a blog post, where we illustrate why memory-based approaches like HNSW fall short in scalability and why disk-based solutions offer a more practical alternative for massive datasets.

[gaocegege]

cc @VoVAllen

We need someone with in-depth expertise in IVF+RabitQ to write the draft. I could help reorg / refine it.