PERFORMANCE: Providing pre-generated index as an argument

[HenrikBengtsson]

I have a huge, static 70 GB tab-delimited file that I want to slice and dice now and then and in the future. Because it's so big, indexing alone will take 10-15 minutes when calling read_tsv(). Since the file is static, this index will be identical every time I try to parse this file, e.g. in a future R session. Is it possible to save the index to file so that it can be imported quickly next time? Something like:

## Indexing takes 10-15 minues
idxs <- index_file(pathname)

## Cache to file
saveRDS(idxs, sprintf("%s.readr_index", pathname))

In future R session:

## Read cached file index
idxs <- readRDS(sprintf("%s.readr_index", pathname))

## Quickly read 100 entries starting from row 10 million.
data <- read_tsv(pathname, index = index, skip = 10e6-1, n_max = 100)

[jimhester]

Not currently possible, however this use case would be more compelling if we had support for memory mapped indexes (tidyverse/vroom#51).

I have experimented with this in branches with good results, but it would still require more work to fully implement.

[rdmorin]

This issue is marked as resolved but I can't find anything in the read_delim or vroom documentation about supplying an existing index to enhance read performance. I was wondering if someone could point me in the right direction?

[jennybc]

I think this is closed here because, if it happened, it would happen in vroom, which is the default backend for readr now.

And the issue tracking it in vroom is the one linked above (tidyverse/vroom#51).

There's no immediate plan to work on this because the effort needed exceeds the developer time dedicated to readr/vroom right now: #1316.