why indexing so slow?

[Wenze7]

I use the following script to index

I have 160G jsonl file, and the number of thread is 24, but the speed is so slow, the following task have ran 5 hour, but only 300K docs has been processed. I want to ask how to speed?

[lintool]

How many files do you have? With 24 threads, you need to have at least 24 separate files to take advantage of the parallelism, i.e., each thread works on a file independently.

[Wenze7]

Thanks! And i have some new problems, I put many *.jsonl file in one dir to index instead of *.jsonl.gz, I'm not sure if it can work?
And in the following log, why so many lines to show 0 documents indexed?

[lintool]

No, what you see is normal. The indexer does not update stats until it's completed files.

[Wenze7]

Thanks, I put many *.jsonl file in one dir to index instead of *.jsonl.gz, I'm not sure if it can work?

[lintool]

Yes, the indexer will handle both .jsonl and .jsonl.gz files. Compression is preferred to save space.