Incremental indexing?

[vishaal27]

Hey, thanks for the great implementation. Are there any plans to add an incremental index construction for large documents? This would be useful when we cannot store the entire corpus in memory, and want to continually corpuses to the constructed index?

[jxmorris12]

Hi @vishaal27 -- can you describe a little bit about how this would work? It's a situation where you can't store the corpus in memory but can store the bm25 vectors in memory, is that right? Because otherwise I think the library as implemented would fail, since the entire BM25 weights are stored as a single tensor.

[vishaal27]

Yes exactly, this is the use-case I was imagining. Thanks!