Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: ChunkedArray uses into_canonical for to_canonical #2503

Closed
wants to merge 5 commits into from
Closed

feat: ChunkedArray uses into_canonical for to_canonical #2503

wants to merge 5 commits into from

Conversation

danking
Copy link
Member

@danking danking commented Feb 25, 2025

No description provided.

@danking danking added the benchmark Run benchmarks on this branch label Feb 25, 2025
@github-actions github-actions bot removed the benchmark Run benchmarks on this branch label Feb 25, 2025
@github-actions GitHub Actions
Copy link
Contributor

Benchmarks: random_access

Table of Results
name PR ca308dd base 1088582 ratio (PR/base) unit
random-access/vortex-tokio-local-disk 2872749 2394121 1.19992 ns
random-access/parquet-tokio-local-disk 253766088 276791143 0.916814 ns

@codspeed-hq CodSpeed HQ
Copy link

codspeed-hq bot commented Feb 25, 2025 •
edited
Loading

CodSpeed Performance Report

Merging #2503 will degrade performances by 18.33%

Comparing dk/chunked-array-into-canonical-canonical-into (06023f0) with develop (04d9d8a)

Summary

⚡ 104 improvements
❌ 1 regressions
✅ 660 untouched benchmarks

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Benchmarks breakdown

Benchmark BASE HEAD Change
⚡ chunked_dict_fsst_into_canonical[(1000, 10, 10)] 1.3 ms 1.2 ms +12.96%
⚡ chunked_dict_fsst_into_canonical[(1000, 10, 100)] 12.9 ms 11.5 ms +11.68%
⚡ chunked_dict_fsst_into_canonical[(1000, 100, 10)] 1.3 ms 1.2 ms +12.92%
⚡ chunked_dict_fsst_into_canonical[(1000, 100, 100)] 12.9 ms 11.6 ms +11.57%
⚡ chunked_dict_fsst_into_canonical[(1000, 1000, 10)] 1.3 ms 1.2 ms +13.26%
⚡ chunked_dict_fsst_into_canonical[(1000, 1000, 100)] 12.9 ms 11.6 ms +11.82%
⚡ chunked_dict_primitive_into_canonical[f32, (1000, 10, 10)] 176.4 µs 86.2 µs ×2
⚡ chunked_dict_primitive_into_canonical[f32, (1000, 10, 100)] 1,540.8 µs 712.8 µs ×2.2
⚡ chunked_dict_primitive_into_canonical[f32, (1000, 100, 10)] 176.4 µs 88.1 µs ×2
⚡ chunked_dict_primitive_into_canonical[f32, (1000, 100, 100)] 1,555 µs 729.1 µs ×2.1
⚡ chunked_dict_primitive_into_canonical[f32, (1000, 1000, 10)] 194 µs 104.6 µs +85.39%
⚡ chunked_dict_primitive_into_canonical[f32, (1000, 1000, 100)] 1,703.1 µs 892.3 µs +90.87%
⚡ chunked_dict_primitive_into_canonical[f64, (1000, 10, 10)] 203.7 µs 104.8 µs +94.38%
⚡ chunked_dict_primitive_into_canonical[f64, (1000, 10, 100)] 1,814.6 µs 896.9 µs ×2
⚡ chunked_dict_primitive_into_canonical[f64, (1000, 100, 10)] 209.6 µs 108 µs +94.08%
⚡ chunked_dict_primitive_into_canonical[f64, (1000, 100, 100)] 1,848.4 µs 929.3 µs +98.91%
⚡ chunked_dict_primitive_into_canonical[f64, (1000, 1000, 10)] 243.2 µs 140.8 µs +72.76%
⚡ chunked_dict_primitive_into_canonical[f64, (1000, 1000, 100)] 2.2 ms 1.3 ms +73.89%
⚡ chunked_dict_primitive_into_canonical[u32, (1000, 10, 10)] 176.7 µs 86.1 µs ×2.1
⚡ chunked_dict_primitive_into_canonical[u32, (1000, 10, 100)] 1,540.7 µs 712.9 µs ×2.2
... ... ... ... ...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

@github-actions GitHub Actions
Copy link
Contributor

Benchmarks: TPC-H on S3

Table of Results
name PR ca308dd base 1088582 ratio (PR/base) unit
tpch_q01/parquet 303980426 3.0005e+08 1.0131 ns
tpch_q02/parquet 796576990 7.56871e+08 1.05246 ns
tpch_q03/parquet 458146044 4.96602e+08 0.922563 ns
tpch_q04/parquet 264890402 2.85126e+08 0.929028 ns
tpch_q05/parquet 640045739 6.17067e+08 1.03724 ns
tpch_q06/parquet 195059226 1.90661e+08 1.02307 ns
tpch_q07/parquet 701306730 7.13328e+08 0.983147 ns
tpch_q08/parquet 823603854 8.97411e+08 0.917756 ns
tpch_q09/parquet 747008411 7.10531e+08 1.05134 ns
tpch_q10/parquet 646926207 6.01207e+08 1.07605 ns
tpch_q11/parquet 301750027 3.20663e+08 0.94102 ns
tpch_q12/parquet 353212044 3.11081e+08 1.13543 ns
tpch_q13/parquet 440474996 4.44059e+08 0.991929 ns
tpch_q14/parquet 282834541 2.87535e+08 0.983654 ns
tpch_q15/parquet 517266552 5.13653e+08 1.00704 ns
tpch_q16/parquet 298757391 2.98801e+08 0.999853 ns
tpch_q17/parquet 432594545 4.4934e+08 0.962732 ns
tpch_q18/parquet 608910405 6.1931e+08 0.983207 ns
tpch_q19/parquet 319274082 3.22417e+08 0.990253 ns
tpch_q20/parquet 575642068 5.62879e+08 1.02267 ns
tpch_q21/parquet 739377926 7.46155e+08 0.990917 ns
tpch_q22/parquet 316034991 2.95304e+08 1.0702 ns
tpch_q01/vortex-file-compressed 302038840 3.20683e+08 0.941862 ns
tpch_q02/vortex-file-compressed 468678139 4.4919e+08 1.04339 ns
tpch_q03/vortex-file-compressed 450999031 4.64773e+08 0.970364 ns
tpch_q04/vortex-file-compressed 396220138 4.01749e+08 0.986239 ns
tpch_q05/vortex-file-compressed 489781301 5.04942e+08 0.969975 ns
tpch_q06/vortex-file-compressed 405817270 4.27909e+08 0.948372 ns
tpch_q07/vortex-file-compressed 619856979 6.25265e+08 0.99135 ns
tpch_q08/vortex-file-compressed 753304518 7.41551e+08 1.01585 ns
tpch_q09/vortex-file-compressed 649716353 6.66385e+08 0.974987 ns
tpch_q10/vortex-file-compressed 547530041 5.32949e+08 1.02736 ns
tpch_q11/vortex-file-compressed 180823162 1.72188e+08 1.05015 ns
tpch_q12/vortex-file-compressed 512372580 5.11439e+08 1.00183 ns
tpch_q13/vortex-file-compressed 161160077 1.6624e+08 0.969439 ns
tpch_q14/vortex-file-compressed 300250250 3.34601e+08 0.897339 ns
tpch_q15/vortex-file-compressed 740523096 7.50867e+08 0.986224 ns
tpch_q16/vortex-file-compressed 207597048 1.98633e+08 1.04513 ns
tpch_q17/vortex-file-compressed 395731820 3.90182e+08 1.01422 ns
tpch_q18/vortex-file-compressed 440280781 4.63244e+08 0.95043 ns
tpch_q19/vortex-file-compressed 447294040 4.59686e+08 0.973043 ns
tpch_q20/vortex-file-compressed 525134352 5.08506e+08 1.0327 ns
tpch_q21/vortex-file-compressed 947373161 1.01454e+09 0.9338 ns
tpch_q22/vortex-file-compressed 175763071 1.68927e+08 1.04047 ns

@github-actions GitHub Actions
Copy link
Contributor

Benchmarks: TPC-H on NVME

Table of Results
name PR ca308dd base 1088582 ratio (PR/base) unit
tpch_q01/arrow 87907450 7.86593e+07 1.11757 ns
tpch_q02/arrow 44101968 4.2061e+07 1.04852 ns
tpch_q03/arrow 36947930 3.45243e+07 1.0702 ns
tpch_q04/arrow 33263040 3.0714e+07 1.08299 ns
tpch_q05/arrow 61114979 5.63448e+07 1.08466 ns
tpch_q06/arrow 9437367 8.31605e+06 1.13484 ns
tpch_q07/arrow 105731001 9.90947e+07 1.06697 ns
tpch_q08/arrow 61405623 5.78342e+07 1.06175 ns
tpch_q09/arrow 93414270 8.39791e+07 1.11235 ns
tpch_q10/arrow 59070970 5.62895e+07 1.04941 ns
tpch_q11/arrow 26476056 2.49011e+07 1.06325 ns
tpch_q12/arrow 34608132 3.12856e+07 1.1062 ns
tpch_q13/arrow 27143722 2.50574e+07 1.08326 ns
tpch_q14/arrow 13489030 1.19784e+07 1.12611 ns
tpch_q15/arrow 28723421 2.644e+07 1.08636 ns
tpch_q16/arrow 23580171 2.18143e+07 1.08095 ns
tpch_q17/arrow 90886476 8.04762e+07 1.12936 ns
tpch_q18/arrow 177963347 1.60646e+08 1.1078 ns
tpch_q19/arrow 28659469 2.59013e+07 1.10649 ns
tpch_q20/arrow 40378793 3.71809e+07 1.08601 ns
tpch_q21/arrow 169176807 1.57494e+08 1.07418 ns
tpch_q22/arrow 16857663 1.72179e+07 0.97908 ns
tpch_q01/parquet 158794968 1.43755e+08 1.10462 ns
tpch_q02/parquet 104870006 9.66266e+07 1.08531 ns
tpch_q03/parquet 109786754 1.10687e+08 0.991863 ns
tpch_q04/parquet 61778375 6.06942e+07 1.01786 ns
tpch_q05/parquet 120175201 1.18341e+08 1.0155 ns
tpch_q06/parquet 28981649 2.95499e+07 0.980768 ns
tpch_q07/parquet 154522509 1.45754e+08 1.06016 ns
tpch_q08/parquet 162649697 1.50988e+08 1.07724 ns
tpch_q09/parquet 211744281 1.94701e+08 1.08754 ns
tpch_q10/parquet 168127212 1.52977e+08 1.09904 ns
tpch_q11/parquet 59590504 4.76757e+07 1.24991 ns
tpch_q12/parquet 105634844 9.96236e+07 1.06034 ns
tpch_q13/parquet 211926343 1.81025e+08 1.1707 ns
tpch_q14/parquet 52283877 5.10167e+07 1.02484 ns
tpch_q15/parquet 99049209 8.00042e+07 1.23805 ns
tpch_q16/parquet 47310571 4.53928e+07 1.04225 ns
tpch_q17/parquet 168416754 1.42934e+08 1.17829 ns
tpch_q18/parquet 280237215 2.36082e+08 1.18703 ns
tpch_q19/parquet 83452983 8.2527e+07 1.01122 ns
tpch_q20/parquet 106527302 9.87186e+07 1.0791 ns
tpch_q21/parquet 224216626 2.08434e+08 1.07572 ns
tpch_q22/parquet 50378644 5.02497e+07 1.00257 ns
tpch_q01/vortex-file-compressed 62813447 5.77062e+07 1.0885 ns
tpch_q02/vortex-file-compressed 51810095 4.8487e+07 1.06853 ns
tpch_q03/vortex-file-compressed 36437831 3.28455e+07 1.10937 ns
tpch_q04/vortex-file-compressed 23528125 2.09891e+07 1.12097 ns
tpch_q05/vortex-file-compressed 59855448 5.46277e+07 1.0957 ns
tpch_q06/vortex-file-compressed 12598605 1.11116e+07 1.13382 ns
tpch_q07/vortex-file-compressed 98265525 8.66663e+07 1.13384 ns
tpch_q08/vortex-file-compressed 67484735 6.19312e+07 1.08967 ns
tpch_q09/vortex-file-compressed 99474199 8.84322e+07 1.12486 ns
tpch_q10/vortex-file-compressed 60440345 5.41484e+07 1.1162 ns
tpch_q11/vortex-file-compressed 28236292 2.55108e+07 1.10684 ns
tpch_q12/vortex-file-compressed 33536698 3.04719e+07 1.10058 ns
tpch_q13/vortex-file-compressed 33286610 3.02597e+07 1.10003 ns
tpch_q14/vortex-file-compressed 16452946 1.45053e+07 1.13427 ns
tpch_q15/vortex-file-compressed 32131547 2.70799e+07 1.18655 ns
tpch_q16/vortex-file-compressed 26751071 2.5494e+07 1.04931 ns
tpch_q17/vortex-file-compressed 87685473 7.65804e+07 1.14501 ns
tpch_q18/vortex-file-compressed 158390138 1.4378e+08 1.10162 ns
tpch_q19/vortex-file-compressed 33956710 3.21661e+07 1.05567 ns
tpch_q20/vortex-file-compressed 44139541 3.87611e+07 1.13876 ns
tpch_q21/vortex-file-compressed 137275210 1.24595e+08 1.10177 ns
tpch_q22/vortex-file-compressed 33831067 2.85816e+07 1.18366 ns

@github-actions GitHub Actions
Copy link
Contributor

Benchmarks: Clickbench on NVME

Table of Results
name PR ca308dd base 1088582 ratio (PR/base) unit
clickbench_q00/parquet 2185479 2.21361e+06 0.987291 ns
clickbench_q01/parquet 60565387 6.18668e+07 0.978964 ns
clickbench_q02/parquet 118489946 1.23406e+08 0.960165 ns
clickbench_q03/parquet 84958581 9.03231e+07 0.940608 ns
clickbench_q04/parquet 638414266 6.90753e+08 0.92423 ns
clickbench_q05/parquet 692842577 7.37386e+08 0.939593 ns
clickbench_q06/parquet 2226313 2.50124e+06 0.890083 ns
clickbench_q07/parquet 62610204 6.81806e+07 0.918299 ns
clickbench_q08/parquet 725444060 7.68246e+08 0.944286 ns
clickbench_q09/parquet 1021388820 1.07353e+09 0.951434 ns
clickbench_q10/parquet 262119297 2.74526e+08 0.954806 ns
clickbench_q11/parquet 313022566 3.25072e+08 0.962932 ns
clickbench_q12/parquet 746349155 7.64285e+08 0.976532 ns
clickbench_q13/parquet 995871275 1.06883e+09 0.931736 ns
clickbench_q14/parquet 739555455 7.62809e+08 0.969516 ns
clickbench_q15/parquet 736009562 7.80245e+08 0.943306 ns
clickbench_q16/parquet 1619429233 1.66023e+09 0.975426 ns
clickbench_q17/parquet 1407769568 1.48566e+09 0.947573 ns
clickbench_q18/parquet 2995483815 3.2595e+09 0.919001 ns
clickbench_q19/parquet 69084524 7.35268e+07 0.939583 ns
clickbench_q20/parquet 1154937319 1.1802e+09 0.978597 ns
clickbench_q21/parquet 1332301236 1.31735e+09 1.01135 ns
clickbench_q22/parquet 1913020776 1.97296e+09 0.969617 ns
clickbench_q23/parquet 7694295563 7.93742e+09 0.96937 ns
clickbench_q24/parquet 439418849 4.59123e+08 0.957083 ns
clickbench_q25/parquet 394639362 3.97222e+08 0.993498 ns
clickbench_q26/parquet 499330631 5.06121e+08 0.986583 ns
clickbench_q27/parquet 1649036990 1.60636e+09 1.02657 ns
clickbench_q28/parquet 11344329516 1.15082e+10 0.98576 ns
clickbench_q29/parquet 440326358 4.56613e+08 0.964332 ns
clickbench_q30/parquet 699802707 7.16453e+08 0.97676 ns
clickbench_q31/parquet 745721698 7.56195e+08 0.98615 ns
clickbench_q32/parquet 2746866219 2.90188e+09 0.946581 ns
clickbench_q33/parquet 2858425043 3.03488e+09 0.941856 ns
clickbench_q34/parquet 2796768922 2.90854e+09 0.961573 ns
clickbench_q35/parquet 846051050 9.11213e+08 0.928489 ns
clickbench_q36/parquet 176505396 1.92213e+08 0.918282 ns
clickbench_q37/parquet 86469032 9.38138e+07 0.921709 ns
clickbench_q38/parquet 110149392 1.15543e+08 0.953322 ns
clickbench_q39/parquet 315798516 3.37624e+08 0.935357 ns
clickbench_q40/parquet 54786524 5.69936e+07 0.961275 ns
clickbench_q41/parquet 52761325 5.54715e+07 0.951142 ns
clickbench_q42/parquet 69692084 7.30304e+07 0.954288 ns
clickbench_q00/vortex-file-compressed 4814265 4.64749e+06 1.03589 ns
clickbench_q01/vortex-file-compressed 23160600 2.40573e+07 0.962727 ns
clickbench_q02/vortex-file-compressed 43006221 4.49179e+07 0.957441 ns
clickbench_q03/vortex-file-compressed 62231557 6.46015e+07 0.963314 ns
clickbench_q04/vortex-file-compressed 565235059 6.13079e+08 0.92196 ns
clickbench_q05/vortex-file-compressed 599576848 6.3388e+08 0.945883 ns
clickbench_q06/vortex-file-compressed 4915632 5.0663e+06 0.97026 ns
clickbench_q07/vortex-file-compressed 28404057 2.85043e+07 0.996482 ns
clickbench_q08/vortex-file-compressed 668118981 7.02705e+08 0.950782 ns
clickbench_q09/vortex-file-compressed 771786933 8.24973e+08 0.93553 ns
clickbench_q10/vortex-file-compressed 114546568 1.1795e+08 0.971146 ns
clickbench_q11/vortex-file-compressed 131679258 1.36694e+08 0.963312 ns
clickbench_q12/vortex-file-compressed 470679899 4.84281e+08 0.971915 ns
clickbench_q13/vortex-file-compressed 697458856 7.19725e+08 0.969063 ns
clickbench_q14/vortex-file-compressed 434597991 4.59973e+08 0.944834 ns
clickbench_q15/vortex-file-compressed 661111828 7.17274e+08 0.921701 ns
clickbench_q16/vortex-file-compressed 1364811000 1.45056e+09 0.940883 ns
clickbench_q17/vortex-file-compressed 1276383562 1.41205e+09 0.903921 ns
clickbench_q18/vortex-file-compressed 2695531871 2.91637e+09 0.924277 ns
clickbench_q19/vortex-file-compressed 35122139 3.60771e+07 0.973531 ns
clickbench_q20/vortex-file-compressed 714462590 7.96077e+08 0.897479 ns
clickbench_q21/vortex-file-compressed 803696490 8.5594e+08 0.938964 ns
clickbench_q22/vortex-file-compressed 1094800048 1.17445e+09 0.932184 ns
clickbench_q23/vortex-file-compressed 1975682862 2.07544e+09 0.951933 ns
clickbench_q24/vortex-file-compressed 174814833 1.81448e+08 0.963441 ns
clickbench_q25/vortex-file-compressed 194557544 1.90812e+08 1.01963 ns
clickbench_q26/vortex-file-compressed 233164232 2.34439e+08 0.994561 ns
clickbench_q27/vortex-file-compressed 1216062514 1.23335e+09 0.985986 ns
clickbench_q28/vortex-file-compressed 10314679934 1.02277e+10 1.0085 ns
clickbench_q29/vortex-file-compressed 716077701 7.13927e+08 1.00301 ns
clickbench_q30/vortex-file-compressed 387159443 3.95055e+08 0.980014 ns
clickbench_q31/vortex-file-compressed 393727215 4.07326e+08 0.966614 ns
clickbench_q32/vortex-file-compressed 2633740855 2.69956e+09 0.975618 ns
clickbench_q33/vortex-file-compressed 2366406378 2.55963e+09 0.92451 ns
clickbench_q34/vortex-file-compressed 2357224635 2.43208e+09 0.96922 ns
clickbench_q35/vortex-file-compressed 914079047 9.40033e+08 0.97239 ns
clickbench_q36/vortex-file-compressed 97363996 1.0038e+08 0.969956 ns
clickbench_q37/vortex-file-compressed 55592161 5.50562e+07 1.00973 ns
clickbench_q38/vortex-file-compressed 39336091 4.18499e+07 0.939932 ns
clickbench_q39/vortex-file-compressed 165972723 1.67522e+08 0.990752 ns
clickbench_q40/vortex-file-compressed 32327761 3.25402e+07 0.993472 ns
clickbench_q41/vortex-file-compressed 31067340 3.17317e+07 0.979063 ns
clickbench_q42/vortex-file-compressed 49411218 5.14423e+07 0.960517 ns

@danking
Copy link
Member Author

danking commented Feb 25, 2025

TPCH S3, parquet ratio: 1.00, vortex ratio: 0.99. Q14 and Q21 stand out as probably real:

name PR base ratio unit
tpch_q14/vortex-file-compressed 300250250 3.34601e+08 0.897339 ns
tpch_q21/vortex-file-compressed 947373161 1.01454e+09 0.9338 ns

TPCH NVME: arrow ratio: 1.08, parquet ratio: 1.08, vortex ratio: 1.11. Maybe a real degradation?

Clickbench NVME: parquet ratio: 0.96, vortex ratio: 0.96

@github-actions GitHub Actions
Copy link
Contributor

Benchmarks: compress

Table of Results
name PR ca308dd base 1088582 ratio (PR/base) unit
compress time/taxi throughput 0.227231 0.217325 1.04558 bytes/ns
parquet_rs-zstd compress time/taxi throughput 0.276716 0.270744 1.02206 bytes/ns
decompress time/taxi throughput 1.75697 1.62693 1.07993 bytes/ns
parquet_rs-zstd decompress time/taxi throughput 1.64345 1.6194 1.01485 bytes/ns
compress time/AirlineSentiment throughput 0.00263223 0.00261565 1.00634 bytes/ns
parquet_rs-zstd compress time/AirlineSentiment throughput 0.0534639 0.0553709 0.965558 bytes/ns
decompress time/AirlineSentiment throughput 0.0233366 0.0252649 0.923674 bytes/ns
parquet_rs-zstd decompress time/AirlineSentiment throughput 0.102737 0.101533 1.01186 bytes/ns
compress time/Arade throughput 0.113531 0.119513 0.949945 bytes/ns
parquet_rs-zstd compress time/Arade throughput 0.396882 0.382052 1.03882 bytes/ns
decompress time/Arade throughput 2.07841 1.89971 1.09407 bytes/ns
parquet_rs-zstd decompress time/Arade throughput 1.90266 1.86969 1.01763 bytes/ns
compress time/Bimbo throughput 0.377802 0.346626 1.08994 bytes/ns
parquet_rs-zstd compress time/Bimbo throughput 0.334107 0.321276 1.03994 bytes/ns
decompress time/Bimbo throughput 2.09657 1.84321 1.13746 bytes/ns
parquet_rs-zstd decompress time/Bimbo throughput 2.83462 2.78835 1.01659 bytes/ns
compress time/CMSprovider throughput 0.0236936 0.0319095 0.742522 bytes/ns
parquet_rs-zstd compress time/CMSprovider throughput 0.353825 0.342213 1.03393 bytes/ns
decompress time/CMSprovider throughput 3.56632 3.37259 1.05744 bytes/ns
parquet_rs-zstd decompress time/CMSprovider throughput 1.81692 1.74198 1.04302 bytes/ns
compress time/Euro2016 throughput 0.138912 0.141446 0.982088 bytes/ns
parquet_rs-zstd compress time/Euro2016 throughput 0.301968 0.295521 1.02181 bytes/ns
decompress time/Euro2016 throughput 2.69399 2.49956 1.07779 bytes/ns
parquet_rs-zstd decompress time/Euro2016 throughput 1.00259 0.968487 1.03521 bytes/ns
compress time/Food throughput 0.149001 0.154038 0.967304 bytes/ns
parquet_rs-zstd compress time/Food throughput 0.319635 0.311769 1.02523 bytes/ns
decompress time/Food throughput 5.11716 4.87978 1.04865 bytes/ns
parquet_rs-zstd decompress time/Food throughput 1.60036 1.58459 1.00995 bytes/ns
compress time/HashTags throughput 0.169997 0.175639 0.967879 bytes/ns
parquet_rs-zstd compress time/HashTags throughput 0.814958 0.782834 1.04103 bytes/ns
decompress time/HashTags throughput 5.07555 4.454 1.13955 bytes/ns
parquet_rs-zstd decompress time/HashTags throughput 2.73456 2.5151 1.08726 bytes/ns
compress time/TPC-H l_comment chunked throughput 0.20692 0.202663 1.02101 bytes/ns
parquet_rs-zstd compress time/TPC-H l_comment chunked throughput 0.284255 0.280158 1.01462 bytes/ns
decompress time/TPC-H l_comment chunked throughput 3.09709 2.84733 1.08772 bytes/ns
parquet_rs-zstd decompress time/TPC-H l_comment chunked throughput 1.07435 1.04851 1.02465 bytes/ns
compress time/TPC-H l_comment canonical throughput 0.0113417 0.0152066 0.745838 bytes/ns
parquet_rs-zstd compress time/TPC-H l_comment canonical throughput 0.28595 0.279015 1.02486 bytes/ns
decompress time/TPC-H l_comment canonical throughput 3.1378 2.9307 1.07067 bytes/ns
parquet_rs-zstd decompress time/TPC-H l_comment canonical throughput 1.07195 1.05398 1.01705 bytes/ns
compress time/wide table cols=10 chunks=1 rows=1000 throughput 0.113281 0.120164 0.942724 bytes/ns
parquet_rs-zstd compress time/wide table cols=10 chunks=1 rows=1000 throughput 0.188467 0.186138 1.01251 bytes/ns
decompress time/wide table cols=10 chunks=1 rows=1000 throughput 0.791083 0.567499 1.39398 bytes/ns
parquet_rs-zstd decompress time/wide table cols=10 chunks=1 rows=1000 throughput 0.5124 0.520014 0.985359 bytes/ns
compress time/wide table cols=100 chunks=1 rows=1000 throughput 0.114576 0.113813 1.0067 bytes/ns
parquet_rs-zstd compress time/wide table cols=100 chunks=1 rows=1000 throughput 0.196624 0.170887 1.15061 bytes/ns
decompress time/wide table cols=100 chunks=1 rows=1000 throughput 1.05839 1.01681 1.04089 bytes/ns
parquet_rs-zstd decompress time/wide table cols=100 chunks=1 rows=1000 throughput 0.502627 0.499813 1.00563 bytes/ns
compress time/wide table cols=1000 chunks=1 rows=1000 throughput 0.104151 0.109778 0.948739 bytes/ns
parquet_rs-zstd compress time/wide table cols=1000 chunks=1 rows=1000 throughput 0.170645 0.161866 1.05424 bytes/ns
decompress time/wide table cols=1000 chunks=1 rows=1000 throughput 0.892128 0.778933 1.14532 bytes/ns
parquet_rs-zstd decompress time/wide table cols=1000 chunks=1 rows=1000 throughput 0.467237 0.435225 1.07355 bytes/ns
compress time/wide table cols=10 chunks=50 rows=1000 throughput 0.0434197 0.0630492 0.688664 bytes/ns
parquet_rs-zstd compress time/wide table cols=10 chunks=50 rows=1000 throughput 0.129295 0.126125 1.02513 bytes/ns
decompress time/wide table cols=10 chunks=50 rows=1000 throughput 0.795588 0.691431 1.15064 bytes/ns
parquet_rs-zstd decompress time/wide table cols=10 chunks=50 rows=1000 throughput 0.534866 0.543917 0.983359 bytes/ns
compress time/wide table cols=100 chunks=50 rows=1000 throughput 0.0432885 0.0537394 0.805527 bytes/ns
parquet_rs-zstd compress time/wide table cols=100 chunks=50 rows=1000 throughput 0.130592 0.108931 1.19885 bytes/ns
decompress time/wide table cols=100 chunks=50 rows=1000 throughput 1.14254 1.03333 1.10569 bytes/ns
parquet_rs-zstd decompress time/wide table cols=100 chunks=50 rows=1000 throughput 0.518968 0.517119 1.00358 bytes/ns
compress time/wide table cols=1000 chunks=50 rows=1000 throughput 0.0371069 0.0508679 0.729476 bytes/ns
parquet_rs-zstd compress time/wide table cols=1000 chunks=50 rows=1000 throughput 0.101581 0.0880994 1.15303 bytes/ns
decompress time/wide table cols=1000 chunks=50 rows=1000 throughput 0.887772 0.864922 1.02642 bytes/ns
parquet_rs-zstd decompress time/wide table cols=1000 chunks=50 rows=1000 throughput 0.442903 0.45981 0.96323 bytes/ns
vortex:raw size/taxi 0.124444 0.124444 1
vortex size/taxi 6.15723e+07 6.15723e+07 1
vortex:parquet-zstd size/taxi 1.10027 1.10027 1
vortex:raw size/AirlineSentiment 1.25903 1.25903 1
vortex size/AirlineSentiment 4112 4112 1
vortex:parquet-zstd size/AirlineSentiment 4.25233 4.25233 1
vortex:raw size/Arade 0.257442 0.257442 1
vortex size/Arade 3.05501e+08 3.05501e+08 1
vortex:parquet-zstd size/Arade 1.00035 1.00035 1
vortex:raw size/Bimbo 0.117103 0.117103 1
vortex size/Bimbo 8.37183e+08 8.37183e+08 1
vortex:parquet-zstd size/Bimbo 2.15685 2.15685 1
vortex:raw size/CMSprovider 0.193268 0.193268 1
vortex size/CMSprovider 1.21471e+09 1.21471e+09 1
vortex:parquet-zstd size/CMSprovider 1.57852 1.57852 1
vortex:raw size/Euro2016 0.475868 0.475875 0.999986
vortex size/Euro2016 2.16531e+08 2.16534e+08 0.999986
vortex:parquet-zstd size/Euro2016 1.82127 1.8213 0.999986
vortex:raw size/Food 0.181292 0.181292 1
vortex size/Food 6.10669e+07 6.10669e+07 1
vortex:parquet-zstd size/Food 1.68553 1.68553 1
vortex:raw size/HashTags 0.146128 0.146128 1
vortex size/HashTags 2.79857e+08 2.79857e+08 1
vortex:parquet-zstd size/HashTags 2.08902 2.08902 1
vortex:raw size/TPC-H l_comment chunked 0.419243 0.418656 1.0014
vortex size/TPC-H l_comment chunked 1.04474e+08 1.04328e+08 1.0014
vortex:parquet-zstd size/TPC-H l_comment chunked 1.8349 1.83246 1.00133
vortex:raw size/TPC-H l_comment canonical 0.425442 0.425399 1.0001
vortex size/TPC-H l_comment canonical 1.06016e+08 1.06006e+08 1.0001
vortex:parquet-zstd size/TPC-H l_comment canonical 1.86197 1.86192 1.00003
vortex:raw size/wide table cols=10 chunks=1 rows=1000 0.622482 0.622482 1
vortex size/wide table cols=10 chunks=1 rows=1000 99688 99688 1
vortex:parquet-zstd size/wide table cols=10 chunks=1 rows=1000 1.06636 1.06636 1
vortex:raw size/wide table cols=100 chunks=1 rows=1000 0.620919 0.620919 1
vortex size/wide table cols=100 chunks=1 rows=1000 994288 994288 1
vortex:parquet-zstd size/wide table cols=100 chunks=1 rows=1000 1.06363 1.06363 1
vortex:raw size/wide table cols=1000 chunks=1 rows=1000 0.620763 0.620763 1
vortex size/wide table cols=1000 chunks=1 rows=1000 9.94029e+06 9.94029e+06 1
vortex:parquet-zstd size/wide table cols=1000 chunks=1 rows=1000 1.06336 1.06336 1
vortex:raw size/wide table cols=10 chunks=50 rows=1000 0.597263 0.597263 1
vortex size/wide table cols=10 chunks=50 rows=1000 99688 99688 1
vortex:parquet-zstd size/wide table cols=10 chunks=50 rows=1000 1.06636 1.06636 1
vortex:raw size/wide table cols=100 chunks=50 rows=1000 0.597024 0.597024 1
vortex size/wide table cols=100 chunks=50 rows=1000 994288 994288 1
vortex:parquet-zstd size/wide table cols=100 chunks=50 rows=1000 1.06363 1.06363 1
vortex:raw size/wide table cols=1000 chunks=50 rows=1000 0.597 0.597 1
vortex size/wide table cols=1000 chunks=50 rows=1000 9.94029e+06 9.94029e+06 1
vortex:parquet-zstd size/wide table cols=1000 chunks=50 rows=1000 1.06336 1.06336 1

@danking danking closed this Feb 25, 2025
@danking
Copy link
Member Author

danking commented Feb 25, 2025

Will revisit once #2508 and #2507 merge.

@danking
Copy link
Member Author

danking commented Feb 25, 2025

see #2511

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@danking