How to search the Wikipedia knowledge base using DPR?

[FutureForMe]

Hi~
I want to use DPR to search Wikipedia, but when I use the following code, an error occurred. I would like to ask, if someone can share with me a code that uses DPR to retrieve WIkipedia, thank you very very very much.

from pyserini.search.faiss import FaissSearcher, TctColBertQueryEncoder

encoder = TctColBertQueryEncoder('castorini/tct_colbert-msmarco')
searcher = FaissSearcher('wikipedia-dpr-100w.dpr-single-nq', encoder)

hits = searcher.search('what')

print(hits)

[lintool]

That's an index built with DPR, so you need to use the DPR encoders: https://github.com/castorini/pyserini/blob/master/pyserini/encode/_dpr.py

@manveertamber can you provide more information?

[manveertamber]

This code will work for you. It uses a DprQueryEncoder trained on Natural Questions to encode the query and then searches the corresponding Wikipedia passage 'wikipedia-dpr-100w.dpr-single-nq' dense index using faiss.

from pyserini.search.faiss import FaissSearcher, DprQueryEncoder

encoder = DprQueryEncoder('facebook/dpr-question_encoder-single-nq-base')
searcher = FaissSearcher.from_prebuilt_index('wikipedia-dpr-100w.dpr-single-nq', encoder)

hits = searcher.search('what is a lobster roll')

for i in range(0, 10):
    print(f'{i+1:2} {hits[i].docid:7} {hits[i].score:.5f}')

And you can get the passage text like this.

print(searcher.doc(hits[0].docid).raw())

[FutureForMe]

Hi, the encoder is ok. But the "FaissSearcher.from_prebuilt_index" don't work. I looked at the source code and printed FAISS_INDEX_INFO in prebuilt_index_info.py when "wiki" in FAISS_INDEX_INFO. The result is shown in the figure below.

So, I changed "wikipedia-dpr-100w.dpr-single-nq" to "wikipedia-dpr-multi-bf" and ran successfully.

[manveertamber]

This difference is because the faiss index names were changed as of this commit: b841b53. You are not using the latest version of Pyserini. wikipedia-dpr-multi-bf was changed to wikipedia-dpr-100w.dpr-multi and wikipedia-dpr-single-nq-bf was changed to wikipedia-dpr-100w.dpr-single-nq. Also note that if you wish to use wikipedia-dpr-multi-bf/wikipedia-dpr-100w.dpr-multi then the corresponding DprQueryEncoder is facebook/dpr-question_encoder-multiset-base.

[FutureForMe]

Thank you very much for your reply. I checked the version of my pyserini, the version is 0.21.0 (I think it should be the latest version), and I encountered the above problem under the 0.21.0 version.

[manveertamber]

Sorry yes, looks like you're right. If you have installed using pip then the old naming is correct. We should do a new PyPi release of Pyserini soon.

[FutureForMe]

Thanks for your hard work on Pyserini. I can experiment with the current version of Pyserini and look forward to your updates.