I'm trying to add a new document for an existing index(Clueweb09)

[tomer92808888]

I'm trying to add a new document for an existing index(Clueweb09)

from pyserini import collection, index

collection = collection.Collection('ClueWeb09Collection', 'data/clueweb')
generator = index.Generator('DefaultLuceneDocumentGenerator')

import json

with open('data/jsonl/clueweb.jsonl', 'w') as f:
    for (i, fs) in enumerate(collection):
        doc = next(fs)
        for (j, doc) in enumerate(fs):
            parsed = generator.create_document(doc)
            docid = parsed.get('id')            # FIELD_ID
            contents = parsed.get('contents')   # FIELD_BODY
            f.write(json.dumps({'id': docid, 'contents': contents}) + '\n')

command = (
            f'python -m pyserini.index.lucene  '
            f'--collection JsonCollection  '
            f'--input data/jsonl  '
            f'--index data/index  '
            f'--generator DefaultLuceneDocumentGenerator  '
            f'--threads 4  '
            f'--storeDocvectors --storeContents'
        )

import subprocess
result = subprocess.run(command, shell=True, check=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)


from pyserini.index.lucene import LuceneIndexer

indexer = LuceneIndexer('data/index', append=True)

doc = "testing"
id = "clueweb09-en0000-00-327300000"

indexer.add_doc_dict({"id": id, "contents": doc})

indexer.close()

But I get the error:
bash JavaException: JVM exception occurred: cannot change field "contents" from storeTermVector=true to inconsistent storeTermVector=false java.lang.IllegalArgumentException

When I do:

indexer.add_doc_dict({"id": id, "contents": doc})

1. How to add new doucments that the only thing that I have is the id and the document itself to this index.
2. How to index the clueweb09 without using command? Can it be with a line of code? Or even the collection can be written easier or different?

@lintool Do you know how to fix it or add a new document with the following command?

Thanks

[tomer92808888]

I managed to find the solution:

from pyserini.index.lucene import LuceneIndexer

args = ["-index", "data/index", "-storeDocvectors", "-storeContents"]

indexer = LuceneIndexer(append=True, args=args)

doc = "testing"
id = "clueweb09-en0000-00-327300000"

indexer.add_doc_dict({"id": id, "contents": doc})

indexer.close()