[Enhancement] Support push down aggregate functions in mv rewrite #49979

LiShuMing · 2024-08-19T11:59:01Z

Why I'm doing:

For simple aggregate functions(eg: min/max/sum), we can push down the agg function even if it's not exactly matched:

    // eg:
    // sum(fn(col)) = fn(sum(col))
    // min(fn(col)) = fn(min(col))
    // max(fn(col)) = fn(max(col))

eg:

CREATE TABLE `tbl1` (
  `k1` date,
  `k2` decimal64(18, 2),
  `k3` varchar(255),
  `v1` bigint 
) ENGINE=OLAP 
DUPLICATE KEY(`k1`, `k2`, `k3`)
DISTRIBUTED BY RANDOM
PROPERTIES (
"replication_num" = "1"
);

CREATE MATERIALIZED VIEW `mv1` 
DISTRIBUTED BY RANDOM
REFRESH ASYNC
PROPERTIES (
"replication_num" = "1"
)
AS SELECT k1, k2, k3, sum(v1) from tbl1 group by k1, k2, k3

-- Cannot be rewritten by mv1!
select t1.k1, 
    sum(case when t1.k1 between date_add('2024-07-20', interval -1 month) and  date_add('2024-07-20', interval 1 month) then t1.v1 else 0 end) 
    from tbl1 t1 group by t1.k1

What I'm doing:

Supports to push down min/max/sum aggregate function
- If the argument is a column ref, and operator map contains the column ref, return true
- If the argument is a call operator and contains multi-column refs(Ony IF/CaseWhen is supported), ensure the aggregate column does not appear in the condition clause.
Add more tests about if(cond, val1, val2) and case when rewrite test cases.

Fixes #issue

What type of PR is this:

Does this PR entail a change in behavior?

Yes, this PR will result in a change in behavior.
No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

Interface/UI changes: syntax, type conversion, expression evaluation, display information
Parameter changes: default values, similar parameters but with different default values
Policy changes: use new policy to replace old one, functionality automatically enabled
Feature removed
Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

I have added test cases for my bug fix or my new feature
This pr needs user documentation (for new or modified features or behaviors)
- I have added documentation for my new feature or new function
This is a backport pr

Bugfix cherry-pick branch check:

...n/java/com/starrocks/sql/optimizer/rule/transformation/materialization/EquationRewriter.java

...tarrocks/sql/optimizer/rule/transformation/materialization/AggregateFunctionRollupUtils.java

LiShuMing · 2024-08-26T05:45:23Z

@mergify rebase

mergify · 2024-08-26T05:45:54Z

rebase

✅ Branch has been successfully rebased

Signed-off-by: shuming.li <[email protected]>

sonarcloud · 2024-09-19T15:17:50Z

Quality Gate passed

Issues
18 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

github-actions · 2024-09-19T16:36:19Z

[Java-Extensions Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

github-actions · 2024-09-19T16:36:49Z

[FE Incremental Coverage Report]

✅ pass : 104 / 112 (92.86%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	com/starrocks/sql/optimizer/rule/transformation/materialization/EquationRewriter.java	95	103	92.23%	[202, 263, 278, 286, 294, 300, 317, 336]
🔵	com/starrocks/sql/optimizer/rule/transformation/materialization/AggregateFunctionRollupUtils.java	5	5	100.00%	[]
🔵	com/starrocks/sql/analyzer/QueryAnalyzer.java	4	4	100.00%	[]

github-actions · 2024-09-19T16:37:10Z

[BE Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

satanson · 2024-09-20T08:33:33Z

...tarrocks/sql/optimizer/rule/transformation/materialization/AggregateFunctionRollupUtils.java

+    // if fn is a scalar function, it can be pushed down to mv union rewrite.
+    public static final Map<String, String> MV_REWRITE_PUSH_DOWN_FUNCTION_MAP = ImmutableMap.<String, String>builder()
+            // Functions and rollup functions are the same.
+            .put(FunctionSet.SUM, FunctionSet.SUM)


why not support COUNT?

satanson · 2024-09-20T08:37:33Z

...n/java/com/starrocks/sql/optimizer/rule/transformation/materialization/EquationRewriter.java

+                return rewriter.rewrite(arg0);
+            } else {
+                // if there are many column refs in arg0, the agg column must be the same.
+                if (arg0Call.getFnName().equalsIgnoreCase(FunctionSet.IF)) {


how about other functions: ifnull, nullif, coalesce?

satanson · 2024-09-20T09:01:10Z

...n/java/com/starrocks/sql/optimizer/rule/transformation/materialization/EquationRewriter.java

+            } else {
+                // if there are many column refs in arg0, the agg column must be the same.
+                if (arg0Call.getFnName().equalsIgnoreCase(FunctionSet.IF)) {
+                    if (arg0Call.getChildren().size() != 3) {


IF functions's children num must be 3, no need check！

satanson · 2024-09-20T09:03:53Z

...n/java/com/starrocks/sql/optimizer/rule/transformation/materialization/EquationRewriter.java

+                return null;
+            }
+            // constant operator
+            if (colRefs.size() == 0) {


how about const null operator?
sum(null) seems should be rewritten into 0;
max(null) seems should be rewritten into null;

LiShuMing requested a review from a team as a code owner August 19, 2024 11:59

starrocks-cr bot reviewed Aug 19, 2024

View reviewed changes

...n/java/com/starrocks/sql/optimizer/rule/transformation/materialization/EquationRewriter.java Show resolved Hide resolved

starrocks-cr bot reviewed Aug 19, 2024

View reviewed changes

...tarrocks/sql/optimizer/rule/transformation/materialization/AggregateFunctionRollupUtils.java Show resolved Hide resolved

mergify bot assigned LiShuMing Aug 19, 2024

github-actions bot added the 3.3 label Aug 19, 2024

LiShuMing force-pushed the fix/main/support_push_down_aggregate_functions_mv_rewrite branch 2 times, most recently from e8ef9ad to 3a360a4 Compare August 23, 2024 10:10

LiShuMing force-pushed the fix/main/support_push_down_aggregate_functions_mv_rewrite branch from 248da0f to 39d686a Compare August 26, 2024 05:45

LiShuMing requested a review from a team as a code owner August 27, 2024 05:17

LiShuMing force-pushed the fix/main/support_push_down_aggregate_functions_mv_rewrite branch from 940cb58 to c004596 Compare August 27, 2024 06:03

Support push down aggregate functions in mv rewrite

b6f7885

Signed-off-by: shuming.li <[email protected]>

LiShuMing force-pushed the fix/main/support_push_down_aggregate_functions_mv_rewrite branch from c004596 to b6f7885 Compare September 19, 2024 15:11

satanson reviewed Sep 20, 2024

View reviewed changes

satanson approved these changes Sep 21, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Enhancement] Support push down aggregate functions in mv rewrite #49979

[Enhancement] Support push down aggregate functions in mv rewrite #49979

LiShuMing commented Aug 19, 2024 •

edited

Loading

LiShuMing commented Aug 26, 2024

mergify bot commented Aug 26, 2024

sonarcloud bot commented Sep 19, 2024

github-actions bot commented Sep 19, 2024

github-actions bot commented Sep 19, 2024

github-actions bot commented Sep 19, 2024

satanson Sep 20, 2024

satanson Sep 20, 2024

satanson Sep 20, 2024

satanson Sep 20, 2024

[Enhancement] Support push down aggregate functions in mv rewrite #49979

Are you sure you want to change the base?

[Enhancement] Support push down aggregate functions in mv rewrite #49979

Conversation

LiShuMing commented Aug 19, 2024 • edited Loading

Why I'm doing:

What I'm doing:

What type of PR is this:

Checklist:

Bugfix cherry-pick branch check:

LiShuMing commented Aug 26, 2024

mergify bot commented Aug 26, 2024

✅ Branch has been successfully rebased

sonarcloud bot commented Sep 19, 2024

Quality Gate passed

github-actions bot commented Sep 19, 2024

[Java-Extensions Incremental Coverage Report]

github-actions bot commented Sep 19, 2024

[FE Incremental Coverage Report]

file detail

github-actions bot commented Sep 19, 2024

[BE Incremental Coverage Report]

satanson Sep 20, 2024

Choose a reason for hiding this comment

satanson Sep 20, 2024

Choose a reason for hiding this comment

satanson Sep 20, 2024

Choose a reason for hiding this comment

satanson Sep 20, 2024

Choose a reason for hiding this comment

LiShuMing commented Aug 19, 2024 •

edited

Loading