index_reader.doc_contents(key) return None while index_reader.doc_raw(key) and index_reader.doc(key) can retrieve document

[zhiyuanpeng]

Hi there,

After I build index on my documents with command:

command = f"python -m pyserini.index -collection JsonCollection \
-generator DefaultLuceneDocumentGenerator -threads {threads} \
-input {self.doc_dir} -index {self.index_dir} -storeRaw \
-storePositions -storeDocvectors"

I try to check the analyzed documents by:

from pyserini.index.lucene import IndexReader
index_reader = IndexReader(index_dir)
anserini_indexed = index_reader.doc_contents(key)
anserini_raw = index_reader.doc_raw(key)
anserini_doc = index_reader.doc(key)

anserini_doc and anserini_raw are not None, but anserini_indexed is None. It seems that the command I used to build the index doesn't keep the analyzed documents. I find the options for building index in this page do not have relevant options.

How to keep the analyzed documents when building the index? Thanks.

[lintool]

Try: https://github.com/castorini/pyserini/blob/master/docs/usage-fetch.md

[zhiyuanpeng]

Thanks. I just found it in the java source file.

[zhiyuanpeng]

I just use -storeContents to build the index again and find that index_reader.doc_contents(key) returns the unanalyzed string of the document instead of the analyzed tokens. How to find the analyzed term of each document in the index? Otherwise, I have to call the analyzer to analyze the documents again. Thanks.

[lintool]

Are you sure you've read the docs? https://github.com/castorini/pyserini/blob/master/docs/usage-indexreader.md#how-do-i-access-and-manipulate-term-vectors

Is this what you are looking for?

[zhiyuanpeng]

Thanks, that is what I am looking for.

[zhiyuanpeng]

BTW, it seems that there is a typo in the doc:

term = 'cities'

# Analyze the term.
analyzed = index_reader.analyze(term)
print(f'The analyzed form of "{term}" is "{analyzed[0]}"')

# Skip term analysis:
df, cf = index_reader.get_term_counts(analyzed[0], analyzer=None)
print(f'term "{term}": df={df}, cf={cf}')

as when skip term analysis, term should be passed in instead of analyzed[0]

df, cf = index_reader.get_term_counts(term, analyzer=None)

[lintool]

Can you please send a PR to fix?
Even better, can you please add a test case?

[zhiyuanpeng]

Sure, will do it.