-
|
Hi, I have built an index for 2.7B documents (had to break it into two indices), and it was quite fast (~4 hours). However, the retrieval is not sufficiently fast and might take many seconds for a document of ~50 words (all documents are around that size). I was wondering if this speed is expected (otherwise I'm doing sth wrong), and if there's a way to increase the speed (such as clustering). |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 1 reply
-
|
Yes, retrieval over 2.7B docs is going to be slow. The standard solution is to break down into separate indexes (say, 100M docs each), search in parallel, and the merge the results. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for your answer! So, you suggest that breaking the index and parallelizing the search using python (using |
Beta Was this translation helpful? Give feedback.
Yes, retrieval over 2.7B docs is going to be slow. The standard solution is to break down into separate indexes (say, 100M docs each), search in parallel, and the merge the results.