Adding TPCH benchmarks for Sort Merge Join #10092

comphead · 2024-04-15T23:55:21Z

Which issue does this PR close?

Related to #9846.

Rationale for this change

The idea is to make Sort Merge Join as stable so this PR adds separate TPCH benchmark for SMJ

What changes are included in this PR?

TPCH benches for SMJ

Are these changes tested?

Are there any user-facing changes?

benchmarks/bench.sh

Co-authored-by: Andy Grove <[email protected]>

alamb · 2024-04-16T16:02:07Z

/benchmark

alamb · 2024-04-16T16:03:07Z

benchmarks/src/tpch/run.rs

+
+    /// Whether to disable collection of statistics (and cost based optimizations) or not.
+    #[structopt(short = "j", long = "hash-join", default_value = "true")]
+    prefer_hash_join: BoolDefaultTrue,


I am worried that this might switch the tpch runs to using SMJ by accident (given your comment above). I started some benchmark runs to see if we can get some data one way or the other

Hash join enabled by default, SMJ is a separate key which is not even documented yet as it still in experimental

benchmarks look good to me. ✅

…to dev

github-actions · 2024-04-16T16:28:27Z

Benchmark results

Benchmarks comparing b54adb3 (main) and 301c827 (PR)

Comparing b54adb3 and 301c827
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query        ┃  b54adb3 ┃  301c827 ┃        Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1     │ 290.98ms │ 288.73ms │     no change │
│ QQuery 2     │  39.69ms │  45.13ms │  1.14x slower │
│ QQuery 3     │  59.75ms │  59.91ms │     no change │
│ QQuery 4     │ 107.13ms │  82.04ms │ +1.31x faster │
│ QQuery 5     │ 100.77ms │ 100.78ms │     no change │
│ QQuery 6     │  15.92ms │  16.06ms │     no change │
│ QQuery 7     │ 228.66ms │ 239.71ms │     no change │
│ QQuery 8     │  41.44ms │  41.96ms │     no change │
│ QQuery 9     │ 123.23ms │ 121.53ms │     no change │
│ QQuery 10    │ 108.09ms │ 110.75ms │     no change │
│ QQuery 11    │  50.80ms │  47.92ms │ +1.06x faster │
│ QQuery 12    │  58.45ms │  58.13ms │     no change │
│ QQuery 13    │ 106.19ms │ 109.85ms │     no change │
│ QQuery 14    │  18.58ms │  19.03ms │     no change │
│ QQuery 15    │  31.15ms │  31.63ms │     no change │
│ QQuery 16    │  45.14ms │  46.85ms │     no change │
│ QQuery 17    │ 151.73ms │ 155.55ms │     no change │
│ QQuery 18    │ 564.21ms │ 574.91ms │     no change │
│ QQuery 19    │  60.81ms │  61.28ms │     no change │
│ QQuery 20    │ 115.52ms │ 115.40ms │     no change │
│ QQuery 21    │ 343.84ms │ 361.40ms │  1.05x slower │
│ QQuery 22    │  39.83ms │  39.36ms │     no change │
└──────────────┴──────────┴──────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary      ┃           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (b54adb3)   │ 2701.93ms │
│ Total Time (301c827)   │ 2727.93ms │
│ Average Time (b54adb3) │  122.81ms │
│ Average Time (301c827) │  124.00ms │
│ Queries Faster         │         2 │
│ Queries Slower         │         2 │
│ Queries with No Change │        18 │
└────────────────────────┴───────────┘
--------------------
Benchmark tpch_sf1.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Query        ┃  b54adb3 ┃  301c827 ┃    Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━┩
│ QQuery 1     │ 434.66ms │ 435.43ms │ no change │
│ QQuery 2     │  57.03ms │  57.94ms │ no change │
│ QQuery 3     │ 144.44ms │ 145.26ms │ no change │
│ QQuery 4     │  89.11ms │  88.29ms │ no change │
│ QQuery 5     │ 202.39ms │ 198.19ms │ no change │
│ QQuery 6     │ 108.01ms │ 107.08ms │ no change │
│ QQuery 7     │ 279.45ms │ 285.34ms │ no change │
│ QQuery 8     │ 187.19ms │ 187.27ms │ no change │
│ QQuery 9     │ 284.80ms │ 288.41ms │ no change │
│ QQuery 10    │ 227.98ms │ 230.87ms │ no change │
│ QQuery 11    │  63.10ms │  61.67ms │ no change │
│ QQuery 12    │ 125.06ms │ 125.32ms │ no change │
│ QQuery 13    │ 175.28ms │ 178.49ms │ no change │
│ QQuery 14    │ 126.85ms │ 127.65ms │ no change │
│ QQuery 15    │ 188.52ms │ 187.98ms │ no change │
│ QQuery 16    │  49.40ms │  49.93ms │ no change │
│ QQuery 17    │ 307.92ms │ 302.43ms │ no change │
│ QQuery 18    │ 454.69ms │ 442.54ms │ no change │
│ QQuery 19    │ 228.25ms │ 229.26ms │ no change │
│ QQuery 20    │ 187.59ms │ 192.72ms │ no change │
│ QQuery 21    │ 329.09ms │ 321.63ms │ no change │
│ QQuery 22    │  52.95ms │  51.98ms │ no change │
└──────────────┴──────────┴──────────┴───────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary      ┃           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (b54adb3)   │ 4303.78ms │
│ Total Time (301c827)   │ 4295.66ms │
│ Average Time (b54adb3) │  195.63ms │
│ Average Time (301c827) │  195.26ms │
│ Queries Faster         │         0 │
│ Queries Slower         │         0 │
│ Queries with No Change │        22 │
└────────────────────────┴───────────┘
--------------------
Benchmark tpch_sf10.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Query        ┃   b54adb3 ┃   301c827 ┃    Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━┩
│ QQuery 1     │ 4292.13ms │ 4262.37ms │ no change │
│ QQuery 2     │  515.87ms │  505.40ms │ no change │
│ QQuery 3     │ 1725.58ms │ 1711.93ms │ no change │
│ QQuery 4     │  815.34ms │  815.34ms │ no change │
│ QQuery 5     │ 2199.55ms │ 2183.73ms │ no change │
│ QQuery 6     │ 1041.50ms │ 1026.73ms │ no change │
│ QQuery 7     │ 3739.83ms │ 3767.13ms │ no change │
│ QQuery 8     │ 2492.95ms │ 2470.22ms │ no change │
│ QQuery 9     │ 4099.85ms │ 4073.77ms │ no change │
│ QQuery 10    │ 2570.39ms │ 2549.11ms │ no change │
│ QQuery 11    │  558.14ms │  553.45ms │ no change │
│ QQuery 12    │ 1188.89ms │ 1188.09ms │ no change │
│ QQuery 13    │ 2321.00ms │ 2339.78ms │ no change │
│ QQuery 14    │ 1278.76ms │ 1281.04ms │ no change │
│ QQuery 15    │ 1942.14ms │ 1917.95ms │ no change │
│ QQuery 16    │  522.90ms │  507.68ms │ no change │
│ QQuery 17    │ 5316.24ms │ 5258.67ms │ no change │
│ QQuery 18    │ 6987.48ms │ 7079.92ms │ no change │
│ QQuery 19    │ 2290.95ms │ 2256.00ms │ no change │
│ QQuery 20    │ 2571.21ms │ 2591.63ms │ no change │
│ QQuery 21    │ 4416.86ms │ 4412.33ms │ no change │
│ QQuery 22    │  564.09ms │  550.44ms │ no change │
└──────────────┴───────────┴───────────┴───────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary      ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (b54adb3)   │ 53451.68ms │
│ Total Time (301c827)   │ 53302.70ms │
│ Average Time (b54adb3) │  2429.62ms │
│ Average Time (301c827) │  2422.85ms │
│ Queries Faster         │          0 │
│ Queries Slower         │          0 │
│ Queries with No Change │         22 │
└────────────────────────┴────────────┘

comphead · 2024-04-16T19:40:28Z

Thanks @andygrove and @alamb for the feedback, please have a second look once you have the time

alamb

Thanks @comphead

alamb · 2024-04-16T20:38:18Z

benchmarks/src/tpch/run.rs

+
+    /// Whether to disable collection of statistics (and cost based optimizations) or not.
+    #[structopt(short = "j", long = "hash-join", default_value = "true")]
+    prefer_hash_join: BoolDefaultTrue,


benchmarks look good to me. ✅

viirya · 2024-04-16T20:47:53Z

Can we trigger the SMJ benchmark using the GitHub action too?

alamb · 2024-04-16T21:05:02Z

Can we trigger the SMJ benchmark using the GitHub action too?

The action has tpch hard coded here: https://github.com/apache/arrow-datafusion/blob/b54adb36b855f198b5098fdb3e4cdf1934818efd/.github/workflows/pr_benchmarks.yml#L41-L51

We could probably add SMJ (or maybe add another command like /benchmark smj 🤔

viirya · 2024-04-16T21:17:29Z

We could probably add SMJ (or maybe add another command like /benchmark smj 🤔

Sounds good. We could do it separately.

github-actions · 2024-04-16T21:31:01Z

Benchmark results

Benchmarks comparing b54adb3 (main) and a88b278 (PR)

Comparing b54adb3 and a88b278
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query        ┃  b54adb3 ┃  a88b278 ┃        Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1     │ 379.88ms │ 289.75ms │ +1.31x faster │
│ QQuery 2     │  39.36ms │  39.14ms │     no change │
│ QQuery 3     │  58.79ms │  58.61ms │     no change │
│ QQuery 4     │  78.61ms │  81.87ms │     no change │
│ QQuery 5     │ 100.43ms │  99.37ms │     no change │
│ QQuery 6     │  15.72ms │  15.94ms │     no change │
│ QQuery 7     │ 219.20ms │ 243.19ms │  1.11x slower │
│ QQuery 8     │  41.27ms │  42.99ms │     no change │
│ QQuery 9     │ 119.24ms │ 122.34ms │     no change │
│ QQuery 10    │ 105.93ms │ 105.67ms │     no change │
│ QQuery 11    │  51.24ms │  47.00ms │ +1.09x faster │
│ QQuery 12    │  58.13ms │  58.60ms │     no change │
│ QQuery 13    │ 103.50ms │ 111.19ms │  1.07x slower │
│ QQuery 14    │  18.65ms │  19.22ms │     no change │
│ QQuery 15    │  30.73ms │  31.22ms │     no change │
│ QQuery 16    │  46.27ms │  46.82ms │     no change │
│ QQuery 17    │ 150.58ms │ 155.36ms │     no change │
│ QQuery 18    │ 487.91ms │ 557.44ms │  1.14x slower │
│ QQuery 19    │  60.97ms │  60.56ms │     no change │
│ QQuery 20    │ 116.59ms │ 115.39ms │     no change │
│ QQuery 21    │ 326.37ms │ 341.62ms │     no change │
│ QQuery 22    │  38.96ms │  40.22ms │     no change │
└──────────────┴──────────┴──────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary      ┃           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (b54adb3)   │ 2648.34ms │
│ Total Time (a88b278)   │ 2683.52ms │
│ Average Time (b54adb3) │  120.38ms │
│ Average Time (a88b278) │  121.98ms │
│ Queries Faster         │         2 │
│ Queries Slower         │         3 │
│ Queries with No Change │        17 │
└────────────────────────┴───────────┘
--------------------
Benchmark tpch_sf1.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Query        ┃  b54adb3 ┃  a88b278 ┃    Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━┩
│ QQuery 1     │ 433.04ms │ 434.42ms │ no change │
│ QQuery 2     │  56.08ms │  56.70ms │ no change │
│ QQuery 3     │ 143.33ms │ 144.09ms │ no change │
│ QQuery 4     │  88.84ms │  86.17ms │ no change │
│ QQuery 5     │ 198.76ms │ 196.19ms │ no change │
│ QQuery 6     │ 107.49ms │ 106.27ms │ no change │
│ QQuery 7     │ 280.46ms │ 281.34ms │ no change │
│ QQuery 8     │ 186.67ms │ 189.23ms │ no change │
│ QQuery 9     │ 281.70ms │ 280.59ms │ no change │
│ QQuery 10    │ 229.77ms │ 234.58ms │ no change │
│ QQuery 11    │  63.09ms │  63.08ms │ no change │
│ QQuery 12    │ 125.00ms │ 126.27ms │ no change │
│ QQuery 13    │ 174.29ms │ 181.39ms │ no change │
│ QQuery 14    │ 126.89ms │ 127.61ms │ no change │
│ QQuery 15    │ 187.88ms │ 186.13ms │ no change │
│ QQuery 16    │  49.97ms │  49.04ms │ no change │
│ QQuery 17    │ 304.42ms │ 300.64ms │ no change │
│ QQuery 18    │ 448.13ms │ 446.06ms │ no change │
│ QQuery 19    │ 227.81ms │ 228.22ms │ no change │
│ QQuery 20    │ 186.57ms │ 188.66ms │ no change │
│ QQuery 21    │ 324.79ms │ 317.73ms │ no change │
│ QQuery 22    │  51.48ms │  52.84ms │ no change │
└──────────────┴──────────┴──────────┴───────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary      ┃           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (b54adb3)   │ 4276.46ms │
│ Total Time (a88b278)   │ 4277.24ms │
│ Average Time (b54adb3) │  194.38ms │
│ Average Time (a88b278) │  194.42ms │
│ Queries Faster         │         0 │
│ Queries Slower         │         0 │
│ Queries with No Change │        22 │
└────────────────────────┴───────────┘
--------------------
Benchmark tpch_sf10.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Query        ┃   b54adb3 ┃   a88b278 ┃    Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━┩
│ QQuery 1     │ 4290.63ms │ 4255.46ms │ no change │
│ QQuery 2     │  497.67ms │  486.75ms │ no change │
│ QQuery 3     │ 1693.08ms │ 1689.86ms │ no change │
│ QQuery 4     │  803.88ms │  804.98ms │ no change │
│ QQuery 5     │ 2175.99ms │ 2162.88ms │ no change │
│ QQuery 6     │ 1037.25ms │ 1030.36ms │ no change │
│ QQuery 7     │ 3636.70ms │ 3651.96ms │ no change │
│ QQuery 8     │ 2441.60ms │ 2442.90ms │ no change │
│ QQuery 9     │ 3996.80ms │ 4005.07ms │ no change │
│ QQuery 10    │ 2549.55ms │ 2547.24ms │ no change │
│ QQuery 11    │  557.16ms │  553.28ms │ no change │
│ QQuery 12    │ 1198.33ms │ 1188.93ms │ no change │
│ QQuery 13    │ 2310.80ms │ 2310.29ms │ no change │
│ QQuery 14    │ 1275.43ms │ 1276.29ms │ no change │
│ QQuery 15    │ 1937.97ms │ 1913.23ms │ no change │
│ QQuery 16    │  505.28ms │  503.25ms │ no change │
│ QQuery 17    │ 5205.04ms │ 5164.46ms │ no change │
│ QQuery 18    │ 6861.06ms │ 6862.22ms │ no change │
│ QQuery 19    │ 2300.81ms │ 2250.29ms │ no change │
│ QQuery 20    │ 2544.61ms │ 2538.00ms │ no change │
│ QQuery 21    │ 4336.69ms │ 4312.40ms │ no change │
│ QQuery 22    │  556.38ms │  552.91ms │ no change │
└──────────────┴───────────┴───────────┴───────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary      ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (b54adb3)   │ 52712.72ms │
│ Total Time (a88b278)   │ 52503.02ms │
│ Average Time (b54adb3) │  2396.03ms │
│ Average Time (a88b278) │  2386.50ms │
│ Queries Faster         │          0 │
│ Queries Slower         │          0 │
│ Queries with No Change │         22 │
└────────────────────────┴────────────┘

github-actions · 2024-04-16T21:46:29Z

Benchmark results

Benchmarks comparing b54adb3 (main) and a88b278 (PR)

Comparing b54adb3 and a88b278
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query        ┃  b54adb3 ┃  a88b278 ┃        Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1     │ 289.92ms │ 289.44ms │     no change │
│ QQuery 2     │  41.06ms │  39.76ms │     no change │
│ QQuery 3     │  58.24ms │  58.61ms │     no change │
│ QQuery 4     │  74.71ms │  85.00ms │  1.14x slower │
│ QQuery 5     │  99.12ms │ 100.13ms │     no change │
│ QQuery 6     │  21.06ms │  16.27ms │ +1.29x faster │
│ QQuery 7     │ 248.87ms │ 246.10ms │     no change │
│ QQuery 8     │  41.32ms │  41.49ms │     no change │
│ QQuery 9     │ 121.11ms │ 124.76ms │     no change │
│ QQuery 10    │ 107.35ms │ 107.21ms │     no change │
│ QQuery 11    │  48.52ms │  49.44ms │     no change │
│ QQuery 12    │  58.68ms │  58.16ms │     no change │
│ QQuery 13    │ 107.56ms │ 109.89ms │     no change │
│ QQuery 14    │  18.39ms │  18.79ms │     no change │
│ QQuery 15    │  30.82ms │  31.44ms │     no change │
│ QQuery 16    │  45.59ms │  46.23ms │     no change │
│ QQuery 17    │ 149.43ms │ 152.85ms │     no change │
│ QQuery 18    │ 533.41ms │ 549.81ms │     no change │
│ QQuery 19    │  60.50ms │  61.15ms │     no change │
│ QQuery 20    │ 111.57ms │ 112.42ms │     no change │
│ QQuery 21    │ 335.38ms │ 317.41ms │ +1.06x faster │
│ QQuery 22    │  38.87ms │  39.27ms │     no change │
└──────────────┴──────────┴──────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary      ┃           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (b54adb3)   │ 2641.47ms │
│ Total Time (a88b278)   │ 2655.64ms │
│ Average Time (b54adb3) │  120.07ms │
│ Average Time (a88b278) │  120.71ms │
│ Queries Faster         │         2 │
│ Queries Slower         │         1 │
│ Queries with No Change │        19 │
└────────────────────────┴───────────┘
--------------------
Benchmark tpch_sf1.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Query        ┃  b54adb3 ┃  a88b278 ┃       Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ QQuery 1     │ 438.46ms │ 434.89ms │    no change │
│ QQuery 2     │  56.45ms │  57.37ms │    no change │
│ QQuery 3     │ 141.75ms │ 143.45ms │    no change │
│ QQuery 4     │  86.65ms │  85.86ms │    no change │
│ QQuery 5     │ 198.93ms │ 200.45ms │    no change │
│ QQuery 6     │ 106.77ms │ 108.35ms │    no change │
│ QQuery 7     │ 275.39ms │ 299.99ms │ 1.09x slower │
│ QQuery 8     │ 185.44ms │ 183.51ms │    no change │
│ QQuery 9     │ 290.97ms │ 288.80ms │    no change │
│ QQuery 10    │ 228.28ms │ 225.52ms │    no change │
│ QQuery 11    │  62.46ms │  62.66ms │    no change │
│ QQuery 12    │ 125.14ms │ 124.20ms │    no change │
│ QQuery 13    │ 177.17ms │ 176.77ms │    no change │
│ QQuery 14    │ 126.34ms │ 126.54ms │    no change │
│ QQuery 15    │ 189.12ms │ 188.74ms │    no change │
│ QQuery 16    │  49.31ms │  49.41ms │    no change │
│ QQuery 17    │ 307.29ms │ 303.14ms │    no change │
│ QQuery 18    │ 450.49ms │ 458.96ms │    no change │
│ QQuery 19    │ 229.13ms │ 225.80ms │    no change │
│ QQuery 20    │ 187.45ms │ 187.92ms │    no change │
│ QQuery 21    │ 321.54ms │ 322.77ms │    no change │
│ QQuery 22    │  51.45ms │  52.18ms │    no change │
└──────────────┴──────────┴──────────┴──────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary      ┃           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (b54adb3)   │ 4286.00ms │
│ Total Time (a88b278)   │ 4307.27ms │
│ Average Time (b54adb3) │  194.82ms │
│ Average Time (a88b278) │  195.78ms │
│ Queries Faster         │         0 │
│ Queries Slower         │         1 │
│ Queries with No Change │        21 │
└────────────────────────┴───────────┘
--------------------
Benchmark tpch_sf10.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Query        ┃   b54adb3 ┃   a88b278 ┃       Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ QQuery 1     │ 4268.05ms │ 4256.07ms │    no change │
│ QQuery 2     │  503.20ms │  499.15ms │    no change │
│ QQuery 3     │ 1687.62ms │ 1698.55ms │    no change │
│ QQuery 4     │  809.73ms │  805.97ms │    no change │
│ QQuery 5     │ 2167.97ms │ 2192.70ms │    no change │
│ QQuery 6     │ 1031.46ms │ 1027.26ms │    no change │
│ QQuery 7     │ 3591.34ms │ 3777.45ms │ 1.05x slower │
│ QQuery 8     │ 2456.81ms │ 2450.82ms │    no change │
│ QQuery 9     │ 4049.37ms │ 4038.77ms │    no change │
│ QQuery 10    │ 2534.86ms │ 2521.17ms │    no change │
│ QQuery 11    │  560.30ms │  562.19ms │    no change │
│ QQuery 12    │ 1187.15ms │ 1198.93ms │    no change │
│ QQuery 13    │ 2314.30ms │ 2308.42ms │    no change │
│ QQuery 14    │ 1280.03ms │ 1267.40ms │    no change │
│ QQuery 15    │ 1938.68ms │ 1935.60ms │    no change │
│ QQuery 16    │  505.99ms │  507.05ms │    no change │
│ QQuery 17    │ 5173.85ms │ 5178.82ms │    no change │
│ QQuery 18    │ 6778.76ms │ 6948.66ms │    no change │
│ QQuery 19    │ 2285.06ms │ 2262.37ms │    no change │
│ QQuery 20    │ 2492.25ms │ 2558.18ms │    no change │
│ QQuery 21    │ 4364.68ms │ 4365.83ms │    no change │
│ QQuery 22    │  569.96ms │  563.16ms │    no change │
└──────────────┴───────────┴───────────┴──────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary      ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (b54adb3)   │ 52551.41ms │
│ Total Time (a88b278)   │ 52924.54ms │
│ Average Time (b54adb3) │  2388.70ms │
│ Average Time (a88b278) │  2405.66ms │
│ Queries Faster         │          0 │
│ Queries Slower         │          1 │
│ Queries with No Change │         21 │
└────────────────────────┴────────────┘

comphead · 2024-04-16T22:31:34Z

Thanks all for the reviews 👍

Adding TPCH bencmarks for Sort Merge Join

c7f272c

comphead changed the title ~~Adding TPCH bencmarks for Sort Merge Join~~ Adding TPCH benchmarks for Sort Merge Join Apr 15, 2024

andygrove reviewed Apr 16, 2024

View reviewed changes

benchmarks/bench.sh Outdated Show resolved Hide resolved

Update benchmarks/bench.sh

301c827

Co-authored-by: Andy Grove <[email protected]>

alamb reviewed Apr 16, 2024

View reviewed changes

comphead added 2 commits April 16, 2024 09:06

fix benches

3faeec9

Merge branch 'dev' of https://github.com/comphead/arrow-datafusion in…

7ba1b96

…to dev

comphead mentioned this pull request Apr 16, 2024

fix Sort Merge Join to pass TPCH tests #10100

Closed

comphead added 2 commits April 16, 2024 09:24

fmt

4d671e2

comments

a88b278

alamb approved these changes Apr 16, 2024

View reviewed changes

viirya approved these changes Apr 16, 2024

View reviewed changes

comphead mentioned this pull request Apr 16, 2024

Trigger the Sort Merge Join benchmark using the GitHub action #10109

Open

comphead merged commit 4ad4f90 into apache:main Apr 16, 2024
24 checks passed

comphead mentioned this pull request Apr 17, 2024

[Epic] Remove Sort Merge Join Experimental status #9846

Open

18 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding TPCH benchmarks for Sort Merge Join #10092

Adding TPCH benchmarks for Sort Merge Join #10092

comphead commented Apr 15, 2024

alamb commented Apr 16, 2024

alamb Apr 16, 2024

comphead Apr 16, 2024

alamb Apr 16, 2024

github-actions bot commented Apr 16, 2024

comphead commented Apr 16, 2024

alamb left a comment

alamb Apr 16, 2024

viirya commented Apr 16, 2024

alamb commented Apr 16, 2024

viirya commented Apr 16, 2024

github-actions bot commented Apr 16, 2024

github-actions bot commented Apr 16, 2024

comphead commented Apr 16, 2024

Adding TPCH benchmarks for Sort Merge Join #10092

Adding TPCH benchmarks for Sort Merge Join #10092

Conversation

comphead commented Apr 15, 2024

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

alamb commented Apr 16, 2024

alamb Apr 16, 2024

Choose a reason for hiding this comment

comphead Apr 16, 2024

Choose a reason for hiding this comment

alamb Apr 16, 2024

Choose a reason for hiding this comment

github-actions bot commented Apr 16, 2024

Benchmark results

comphead commented Apr 16, 2024

alamb left a comment

Choose a reason for hiding this comment

alamb Apr 16, 2024

Choose a reason for hiding this comment

viirya commented Apr 16, 2024

alamb commented Apr 16, 2024

viirya commented Apr 16, 2024

github-actions bot commented Apr 16, 2024

Benchmark results

github-actions bot commented Apr 16, 2024

Benchmark results

comphead commented Apr 16, 2024