-
Notifications
You must be signed in to change notification settings - Fork 847
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
chore: Add initial benchmarks for Arrow IPC Stream writer #6972
base: main
Are you sure you want to change the base?
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you @andygrove
I tried this out locally:
cargo bench --bench stream_writer
I also made a flamegraph with
sudo flamegraph -- target/release/deps/stream_writer-a9122e21216257a4 --bench
I have a suggestion for improvement (don't buffer) but that could be done as a follow on
group.bench_function("write_multiple_batches", |b| { | ||
let batch = create_batch(8192, true); | ||
b.iter(|| { | ||
let buffer = vec![]; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Interestingly, when I resized the output buffers reasonably in andygrove#76 and reran the flamegraph, a substantial amount of time is still spent reallocating / resizing: |
I believe this is waiting on andygrove#76 so marking as draft |
Which issue does this PR close?
Part of #6968
Rationale for this change
Add some initial benchmarks to help with efforts to optimize IPC reading and writing.
This is a copy of existing code that we had in Comet and is not as comprehensive as the suggestions in #6968 but it provides a starting point at least.
What changes are included in this PR?
Are there any user-facing changes?