How to load persisted tantivy index?

[JRatschat]

Hi all,

I am not able to find documentation on how to load a persisted tantivy index. Hence, when I use the example in the README.md index = tantivy.Index(schema, path=os.getcwd() + '/index'), how can load the index without specifying the schema or committing documents again?

Thanks!

[cjrh]

Thank you for pointing this out. It's just an oversight in the documentation.

There is an open() staticmethod on the Index class that loads an index from a directory path. You can see this by looking at the code for the Index class:

#[pyclass(module = "tantivy.tantivy")]
pub(crate) struct Index {
    pub(crate) index: tv::Index,
    reader: tv::IndexReader,
}

#[pymethods]
impl Index {
    #[staticmethod]
    fn open(path: &str) -> PyResult<Index> {
        let index = tv::Index::open_in_dir(path).map_err(to_pyerr)?;

        Index::register_custom_text_analyzers(&index);

        let reader = index.reader().map_err(to_pyerr)?;
        Ok(Index { index, reader })
    }

Here's a worked example:

>>> import tantivy
>>> schema_builder = tantivy.SchemaBuilder()
>>> schema_builder.add_text_field("title", stored=True)
<builtins.SchemaBuilder object at 0x759b82088410>

>>> schema_builder.add_text_field("body", stored=True)
<builtins.SchemaBuilder object at 0x759b8477d1f0>

>>> schema = schema_builder.build()
>>> import os
>>> p = os.getcwd() + '/testblah'
>>> index = tantivy.Index(schema, path=p)

>>> writer = index.writer()
>>> writer.add_document(tantivy.Document(
...     doc_id=1,
...     title=["The Old Man and the Sea"],
... ))
0

>>> writer.commit()
2

# Get rid of the identier
>>> del index

# Try to load the index from disk
>>> index = tantivy.Index.open(p)
>>>

One thing to mention, I got "Directory does not exist" errors when I tried to do the step to create index on disk:

>>> import os
>>> p = os.getcwd() + '/testblah'
>>> index = tantivy.Index(schema, path=p)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: Directory does not exist: '/home/caleb/Documents/repos/tantivy-py/testblah'.

I didn't look at this for too long so it's possible I forgot something, but in any case I got around this by creating the dir ./testblah manually with mkdir in the shell.

[JRatschat]

Thanks @cjrh!

This solved my question. I also got the same error ("Directory does not exist") when testing index = tantivy.Index(schema, path=p) earlier - creating the directory manually indeed prevents this error.