Add multi-source reading to JSON reader benchmarks #17688
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
shrshi
changed the title
|
Null probability has been set to zero in the random table generator so that the JSON reader benchmark does not fail for multi-source multi-batch reading. Once issue #17689 is resolved, the change to the table generator will be reverted. |
shrshi
added
cuIO
shrshi
changed the title
|
|
||
| json_read_common(source_sink, num_rows, state, comptype, data_size); | ||
| std::vector<char> out_buffer; | ||
| auto sink = cudf::io::sink_info(&out_buffer); |
There was a problem hiding this comment.
Passing a pointer to a vector in modern C++ is generally odd. Are you referring to out_buffer.data() or just out_buffer?
There was a problem hiding this comment.
I'm using the data sink constructor that accepts a pointer to a output host vector -
cudf/cpp/include/cudf/io/types.hpp
Line 555 in 39bcd16
It is indeed a little awkward to pass a pointer to a vector, but I'm not sure if there's a cleaner way to construct a
sink_info object.@vuule do you have any suggestions on a better approach?
There was a problem hiding this comment.
It seems that the _buffer data member of sink_info should be a vector of char spans, but addressing this is beyond the scope of the current PR.
There was a problem hiding this comment.
How did I miss these comments?
The writer resizes the vector, as the output size is not known in advance. Passing a span unfortunately does not work here.
There was a problem hiding this comment.
Ah OK, then the buffer owner could use a shared pointer to avoid passing raw pointers around.
|
/merge |
Description
Depends on #17708
Enables benchmarking of multi-source multi-batch JSON reader by adding another axis for number of input sources in the benchmark.
Checklist