Support efficient way of retrieving results for multiple path queries

[iilyak]

Given an example JSON I would like to be able to take arbitrary structure and extract multiple paths to return results as a flat dictionary. It can be done outside the rsonpath by calling it multiple times, but it would be really inefficient. It would be way better if we would be able to issue API call and collect the results in one go without traversing the data multiple times.

{
  "person": {
    "name": "John",
    "surname": "Doe",
    "phoneNumbers": [
      {
        "type": "Home",
        "number": "111-222-333"
      },
      {
        "type": "Work",
        "number": "123-456-789"
      }
    ]
  }
}

This is an example of the result

{
   "full_name": "John Doe" // concatenation of two fields
   "work_phone": "123-456-789"
}

The specification of the query could look like (I use JSON for demonstration purposes)

{
    "full_name": "rsonpath-query-to-calculate-full-name",
    "work_phone": "rsonpath-query-to-extract-work-phone"
}

[V0ldek]

Hey, thanks for interest in rsonpath!

Is this a request about the Rust library or are we talking about the rq CLI?

[iilyak]

Hey, thanks for interest in rsonpath!

Is this a request about the Rust library or are we talking about the rq CLI?

Rust library

[iilyak]

Some background. I wanted to contribute a feature into https://github.com/typst/typst which would take a data file (json/yaml) and given a query it would extract arbitrary fields and combinations of them. Then I would use typst language to generate a table and write it into pdf. The problem is that json file would be quite big (hundred of megabytes) but the result of each sub-query expected to be small. To avoid multiple scans (I need to extract about 20-30 different nested fields) I need something like either:

1. ability to run multiple queries and collect the results in the accumulator
2. iterator over json which would be checking matches against multiple queries
3. fold with engine which matches on multiple queries with accumulator

The #1 fits the bill perfectly. The #2 and #3 is kind of similar and essentially the same idea just implemented differently.

[V0ldek]

I wanted to contribute a feature into https://github.com/typst/typst

uu, helping with typst development would be exciting since I use it myself :)

The problem is that json file would be quite big (hundred of megabytes) but the result of each sub-query expected to be small. To avoid multiple scans (I need to extract about 20-30 different nested fields)

Okay, that's neat since it means we can very concretely test our performance against actual usage. I'd start by actually verifying how long running your queries separately takes. Can you tell me how to obtain this hundreds-of-megs JSON file and what queries you will run on it?

Modifying the engine to run multiple queries at the same time into multiple sinks would be feasible (at least as a prototype), but before going for a full design we'd need to measure how that compares against the baseline. There's also the immediate "brutal" solution to the problem, namely running your 30 queries on 30 threads, which has a real chance to be faster than any multi-query single-threaded solution. I'd be happy to experiment with a real JSON and queries :)

The specification of the query could look like (...)

Your idea of having a kind of an rsonpath specification on how to calculate a full result sounds neat, however it'd require some bespoke expression language to express stuff like "concatenate strings", so that's a huge design space. I think this particular issue though is solvable via JSONPath only, in your example the query:

$['person']['name', 'surname']

would give you ["John", "Doe"] and you can concatenate that after running the query. The spec guarantees the order. The issue is ofc that rsonpath doesn't support multiple selectors per segment yet, but that's something that's on the roadmap to support full JSONPath anyway. For now just splitting that into two queries would be a workaround.

Extracting the work phone is also problematic since it requires a filter, but that's something we are actively working on to solve (#156).

[iilyak]

an you tell me how to obtain this hundreds-of-megs JSON file and what queries you will run on it?

I cannot share the JSON I have. However any big JSON would do. You can use one of the available open datasets. For example 340Mb JSON

[V0ldek]

I wrote a quick-and-dirty example of trying to run multiple path queries on the same file but using multithreading.

I run this on the Steam JSON you provided and the following four queries:

• $[*]..requirements.minimum..os
• $[*]..requirements.minimum..processor
• $[*]..requirements.minimum..memory
• $[*]..popu_tags[*]

The runtime on my machine is 77ms (2.02GB/s) on average, which is almost exactly the same as just running the last (heaviest) query, so parallelism is near-perfect.

You can check sequential execution by changing the par_iter line to iter, in which case I get 197ms.

Unless you wish to run hundreds of queries at the same time simple parallalisation will most likely do the trick. Because of how query compilation works, trying to compile two queries into one engine would most likely be slower than just parallelizing them, since it would miss out on our SIMD skipping opportunities. That is at least until we figure out a way to capture filtering in our internal automaton, which could enable better optimizations. But there's a lot of theoretical work we need to do before we can implement that.

Code

```rust use clap::Parser; use rayon::prelude::*; use rsonpath::engine::error::EngineError; use rsonpath::{ engine::{Compiler, Engine, RsonpathEngine}, input::MmapInput, }; use std::sync::Arc; use std::time::Instant; use std::{error::Error, fs, process::ExitCode};

#[derive(Parser)]
struct Args {
file: String,
#[clap(num_args = 1..)]
queries: Vec,
}

fn main() -> Result<ExitCode, Box> {
let args = Args::parse();

let queries = args
    .queries
    .iter()
    .map(|q| rsonpath_syntax::parse(q))
    .collect::<Result<Vec<_>, _>>()?
    .into_iter()
    .map(|q| RsonpathEngine::compile_query(&q))
    .collect::<Result<Vec<_>, _>>()?;
let file = fs::File::open(&args.file)?;
// SAFETY: File is kept open until end of the run.
let input = Arc::new(unsafe { MmapInput::map_file(&file)? });

let start = Instant::now();
let sinks = queries
    .par_iter()
    .map(|q| {
        let mut sink = vec![];
        q.matches(input.as_ref(), &mut sink)?;
        Ok(sink)
    })
    .collect::<Result<Vec<_>, EngineError>>()?;
let end = start.elapsed();

let total_matches: usize = sinks.into_iter().map(|v| v.len()).sum();

println!("{}ms elapsed for {total_matches} total results", end.as_millis());

Ok(ExitCode::SUCCESS)

}

</p>
</details> 

[iilyak]

I was hopping the query engine using something like lenses and can focus on different aspects of a query while traversing the tree once.