Conversion of the internal document ids in Faiss

[galuscakova]

Hi,
can I please ask how can I get the original document id from the id which is used internally in the faiss index?
I found the instructions for the ids conversion using LuceneSearch, but cannot find anything for the Faiss index and dense retrieval.

Thanks!

[MXueguang]

Hi @galuscakova,
Could you describe how would you like to access the original id?
The search results from dense retrieval is already converted from internal id back to original document id.

pyserini/pyserini/search/faiss/_searcher.py

Line 439 in 5b087f1

return [DenseSearchResult(self.docids[idx], score)

[galuscakova]

Thanks for the fast answer!
I am using pre-trained MS Marco passage index v2. The original corpus uses passage ids in following format: msmarco_passage_00_997. But when I run retrieval, I get the numerical ids such as '7973234'.
Is there a way how to get the mapping from these numerical ids to the ones used in the corpus?

[MXueguang]

the numerical id '7973234' seems like the passage id from msmarco-v1-passage. Could you let me know which pre-built index are you using

[galuscakova]

Sure, I believe, I am using the version 2: msmarco-passage-tct_colbert-v2-bf, as in the example: https://github.com/castorini/pyserini/blob/master/docs/experiments-tct_colbert-v2.md
But I guess you might be right -- there are 8841822 documents in the index, what seems to correspond to the number of passages in the version 1. I see the same number in the index-msmarco-passage-tct_colbert-v2-hnp-bf.

[MXueguang]

The above document is for "tct_colbert-v2" on MSMARCO "v1".
I guess this is what you are looking for: https://github.com/castorini/pyserini/blob/master/docs/experiments-msmarco-v2-tct_colbert-v2.md

[galuscakova]

Ah, ok, yes, thanks for your help!