Huge memory usage by cyclestreets::batch_read

[mem48]

Looking into this. Part of the problem is the json from the commute routes is 10 GB out of the box.

results = RcppSimdJson::fparse(results_raw, query = "/marker", query_error_ok = TRUE, always_list = TRUE)

Takes an additional 8GB to run

[Robinlovelace]

I think that's a problem upstream in cyclestreets. Can we set it to ignore all the key-value pairs we're not interested in?

[Robinlovelace]

Seeing as this is a cyclestreets/cyclestreets-r issue, where the fix should be, I'm going to try to migrate it there...

[Robinlovelace]

Heads-up @mem48 it's here now.