Weekly plan 01-30-2025

[XiangpengHao]

The highest priority for me is to get a complete ClickBench with reasonable performance.

• Run entire ClickBench (@XiangpengHao )
• Collect reasonable metrics
• Ensure the results we get are correct (@WetABQ working on this)
• Optimize some of the decoding stuff, as they show up in the flamegraph
• Sign off this task and verify the system behave as expected.

Before Monday: Xiangpeng run and tune entire ClickBench.

Before Wednesday meeting, try to make the Figure 1 in #15

[XiangpengHao]

This is how I check ClickBench results before: https://github.com/XiangpengHao/datafusion/blob/research/benchmarks/src/clickbench.rs#L245

It can be a good starting point, we probably need something more comprehensive than this. @WetABQ

[XiangpengHao]

The goal of this series of todo is to get back the performance to what we present in the talks

My next todo:

• automatically collect a flamegraph after one query
• Cache the metadata in parquet factory
• automatically save stats after a query
• Write benchmark results to file
• Consume batches in cached_array reader will decode multiple times, we should merge them into once.
• merge smaller batches into larger batches before sending over network
• We should never use schema mapper in client
• Disable arrow ipc bounds check
• Introduce a metric system, want to know: how many bytes sent over network, the record batch size distribution, selectivity etc.

Automatically collect:

• Overall latency
• network traffic
• time spent on cache server
• time spent on compute server
• cache memory consumption

[XiangpengHao]

Filed spiraldb/fsst#69, which they fixed in a minute. Will bump our dependencies once it releases

[XiangpengHao]

We are currently blocked by this: apache/arrow-rs#7058

Need to fix it as soon as possible

[XiangpengHao]

We are currently blocked by this: apache/arrow-rs#7058

Need to fix it as soon as possible

Tried to fix it, but failed, the flatbuffer is too difficult to debug... But I find a way to workaround it, so we good now.

[XiangpengHao]

Done

[XiangpengHao]

next: #42