Can the parsing performance be improved by using a precomputed index?

[KOLANICH]

Hi. I use your parser to parse large (in fact not very large, from 200 MiB to 2 GiB each file is) CSV datasets consisting mostly of strings. No DB functionality is needed, so I feel like moving the data into a real DB as a preprocessing step will not bring any benefit.

Also I cache intermediate results into plain text files, one record per a line, but lines are not necessarily CSV.

I have designed a file format (mostly a prefix tree with additional data in each non-leaf node, just not to waste space) allowing one to save the offsets of line breaks for getting a record by its line number fast.

I have also extended it to hold arbitrary shapes. So we can save not only line breaks, but other separators, like commas (and no separators at all). The format seems to be very versatile and multipurpose.

But all of that scannjng code is currently coded in python. Intel Hyperscan (a fast large scale jitted AVX-accelerated regex matching engine used in DPIs) is not useful, since it cannot match subgroups.

I wonder if we can accelerate parsing by first precomputing the index file (it is very likely it can even be done on GPU, though I am not sure if it will bring any benefit) and saving it (done once), then by using the offsets from it (can be done multiple times) to avoid scanning and tokenizing entirely.

[ben-strasser]

If you can create an index, then my recommendation is to work with binary files and read-only mmap. Nothing beats a simple array lookup. This library is meant to parse CSV files that you see for the first time.

I have experimented with SIMD code in the past but was not able to get a consistent significant speedup. For this reason, I stuck with the simpler approach. GPU makes IMO no sense as just the data copying from GPU to CPU will be slower than than the CPU parsing.

[KOLANICH]

If you can create an index, then my recommendation is to work with binary files and read-only mmap.

Yeah, it's the way I use it, but currently it is the for plain index only. I haven't yet finished a trie-based index, because after I added the data for fast arbitrary lookup by element index, the trie-based index is usually only twice smaller than the flat one. But applying the trie compression recursively to this auxillary information looks promising, but I need to tinker bit more around it.

But all of that scanning code is currently coded in python.

is no longer true, I have rewritten the scanner in C++ and paralleled it aggressively and have also implemented JIT for x86_64. https://github.com/ScanBytes/ScanBytes.cpp Very surprisingly, std::thread-based impl turned out to be the fastest, and Intel TBB-based one turned out to be the slowest and OpenMP in between.

GPU makes IMO no sense as just the data copying from GPU to CPU will be slower than than the CPU parsing.

It depends. Some devices have a GPU and a CPU within a single chip and use the same memory.