Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[feature](function) support orthogonal_bitmap_expr_calculate & orthogonal_bitmap_expr_calculate_count for nereids #44991

Merged
merged 4 commits into from
Dec 4, 2024
Merged

[feature](function) support orthogonal_bitmap_expr_calculate & orthogonal_bitmap_expr_calculate_count for nereids #44991

merged 4 commits into from
Dec 4, 2024

Conversation

924060929
Copy link
Contributor

@924060929 924060929 commented Dec 4, 2024 •
edited
Loading

What problem does this PR solve?

support orthogonal_bitmap_expr_calculate & orthogonal_bitmap_expr_calculate_count for nereids

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@924060929 924060929 changed the title [feature](function support orthogonal_bitmap_expr_calculate & orthogonal_bitmap_expr_calculate_count for nereids [feature](function) support orthogonal_bitmap_expr_calculate & orthogonal_bitmap_expr_calculate_count for nereids Dec 4, 2024
@924060929 924060929 force-pushed the orthogonal_bitmap_expr_calculate branch from d2a8f61 to 0411769 Compare December 4, 2024 08:47
@924060929
Copy link
Contributor Author

run buildall

@924060929
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 38.48% (10008/26007)
Line Coverage: 29.52% (83887/284164)
Region Coverage: 28.63% (43125/150607)
Branch Coverage: 25.23% (21916/86882)
Coverage Report: http://coverage.selectdb-in.cc/coverage/97075b7e36c802be7e017dbf98adcfb407e0db5a_97075b7e36c802be7e017dbf98adcfb407e0db5a/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 40186 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 97075b7e36c802be7e017dbf98adcfb407e0db5a, data reload: false

------ Round 1 ----------------------------------
q1	17614	8248	7334	7334
q2	2055	183	184	183
q3	10563	1090	1190	1090
q4	10576	776	811	776
q5	7603	2665	2739	2665
q6	242	153	153	153
q7	1016	630	609	609
q8	9257	1862	1932	1862
q9	6746	6520	6489	6489
q10	6998	2276	2299	2276
q11	481	264	261	261
q12	437	236	218	218
q13	17771	3037	3076	3037
q14	243	207	223	207
q15	567	518	524	518
q16	669	591	595	591
q17	988	627	523	523
q18	7303	6747	6754	6747
q19	1344	1043	949	949
q20	499	186	180	180
q21	4019	3203	3263	3203
q22	376	320	315	315
Total cold run time: 107367 ms
Total hot run time: 40186 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7246	7264	7271	7264
q2	330	232	234	232
q3	2905	2919	2932	2919
q4	2080	1837	1969	1837
q5	5668	5640	5654	5640
q6	236	150	150	150
q7	2235	1817	1808	1808
q8	3419	3580	3509	3509
q9	8950	9152	9022	9022
q10	3606	3542	3555	3542
q11	597	500	510	500
q12	797	613	623	613
q13	13034	3288	3260	3260
q14	308	266	281	266
q15	585	543	536	536
q16	695	664	660	660
q17	1875	1683	1627	1627
q18	8412	7919	7722	7722
q19	1691	1678	1714	1678
q20	2101	1841	1850	1841
q21	5650	5420	5418	5418
q22	658	569	599	569
Total cold run time: 73078 ms
Total hot run time: 60613 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 197415 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 97075b7e36c802be7e017dbf98adcfb407e0db5a, data reload: false

query1	1247	993	952	952
query2	6251	2147	2138	2138
query3	10932	4400	4405	4400
query4	67262	29029	23745	23745
query5	4920	476	463	463
query6	407	182	172	172
query7	5590	304	297	297
query8	323	240	261	240
query9	8654	2650	2647	2647
query10	455	243	254	243
query11	17242	15401	16130	15401
query12	157	111	107	107
query13	1470	422	438	422
query14	9883	7899	6738	6738
query15	226	195	188	188
query16	7422	464	450	450
query17	1127	601	570	570
query18	1846	303	308	303
query19	199	151	149	149
query20	115	109	107	107
query21	245	112	106	106
query22	4812	4450	4700	4450
query23	34947	34613	34498	34498
query24	5370	2549	2614	2549
query25	498	397	391	391
query26	640	151	155	151
query27	1785	284	292	284
query28	4619	2471	2444	2444
query29	672	428	415	415
query30	218	157	163	157
query31	1058	854	863	854
query32	61	56	53	53
query33	453	299	310	299
query34	941	514	521	514
query35	934	767	756	756
query36	1072	947	975	947
query37	154	78	73	73
query38	4498	4433	4467	4433
query39	1521	1502	1485	1485
query40	208	101	98	98
query41	45	43	45	43
query42	119	99	100	99
query43	558	506	512	506
query44	1232	860	833	833
query45	185	168	172	168
query46	1159	733	729	729
query47	2059	1906	1938	1906
query48	436	318	318	318
query49	724	406	397	397
query50	852	388	385	385
query51	7427	7358	7068	7068
query52	98	86	84	84
query53	256	178	174	174
query54	510	397	400	397
query55	79	74	74	74
query56	250	229	230	229
query57	1251	1109	1121	1109
query58	217	212	212	212
query59	3359	2982	2975	2975
query60	271	242	245	242
query61	113	108	102	102
query62	791	665	683	665
query63	208	185	182	182
query64	1435	683	647	647
query65	3263	3215	3215	3215
query66	704	305	305	305
query67	16134	15607	15755	15607
query68	4199	556	559	556
query69	430	265	252	252
query70	1194	1135	1131	1131
query71	363	246	251	246
query72	6531	4081	4049	4049
query73	753	349	370	349
query74	10318	9060	9127	9060
query75	3376	2634	2694	2634
query76	2031	1114	1050	1050
query77	484	266	282	266
query78	10407	9512	9444	9444
query79	1705	592	593	592
query80	1036	427	530	427
query81	503	233	233	233
query82	218	120	121	120
query83	171	144	139	139
query84	282	87	74	74
query85	971	305	315	305
query86	419	284	289	284
query87	4810	4586	4539	4539
query88	3868	2196	2170	2170
query89	426	292	284	284
query90	2059	190	185	185
query91	137	102	105	102
query92	63	50	50	50
query93	2294	547	544	544
query94	914	271	288	271
query95	350	249	249	249
query96	629	284	280	280
query97	2851	2683	2687	2683
query98	219	204	188	188
query99	1591	1330	1299	1299
Total cold run time: 319771 ms
Total hot run time: 197415 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.53 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 97075b7e36c802be7e017dbf98adcfb407e0db5a, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.04
query3	0.23	0.07	0.06
query4	1.64	0.11	0.10
query5	0.44	0.41	0.41
query6	1.18	0.68	0.65
query7	0.03	0.02	0.02
query8	0.04	0.03	0.03
query9	0.59	0.51	0.50
query10	0.55	0.56	0.56
query11	0.14	0.10	0.11
query12	0.13	0.12	0.12
query13	0.62	0.60	0.61
query14	2.73	2.73	2.74
query15	0.91	0.83	0.84
query16	0.38	0.39	0.38
query17	0.97	1.06	1.06
query18	0.22	0.20	0.21
query19	2.00	1.82	2.03
query20	0.01	0.01	0.01
query21	15.36	0.58	0.56
query22	2.53	2.72	1.53
query23	16.93	1.05	0.83
query24	3.00	1.18	1.27
query25	0.28	0.18	0.05
query26	0.54	0.15	0.13
query27	0.04	0.04	0.04
query28	10.38	1.10	1.08
query29	12.53	3.25	3.22
query30	0.25	0.06	0.06
query31	2.86	0.38	0.37
query32	3.32	0.47	0.47
query33	3.01	3.15	3.04
query34	17.09	4.47	4.44
query35	4.58	4.52	4.55
query36	0.68	0.47	0.49
query37	0.09	0.06	0.06
query38	0.04	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.13	0.11
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.77 s
Total hot run time: 32.53 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 4, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Dec 4, 2024

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Dec 4, 2024

PR approved by anyone and no changes requested.

@924060929 924060929 merged commit d58a972 into apache:master Dec 4, 2024
22 of 25 checks passed
924060929 added a commit to 924060929/incubator-doris that referenced this pull request Dec 5, 2024
@924060929
[feature](function) support orthogonal_bitmap_expr_calculate & orthog…
8973f8a 
…onal_bitmap_expr_calculate_count for nereids (apache#44991)

support orthogonal_bitmap_expr_calculate &
orthogonal_bitmap_expr_calculate_count for nereids

(cherry picked from commit d58a972)
@924060929 924060929 mentioned this pull request Dec 5, 2024
Merged
924060929 added a commit to 924060929/incubator-doris that referenced this pull request Dec 5, 2024
@924060929
[feature](function) support orthogonal_bitmap_expr_calculate & orthog…
5492dd3 
…onal_bitmap_expr_calculate_count for nereids (apache#44991)

support orthogonal_bitmap_expr_calculate &
orthogonal_bitmap_expr_calculate_count for nereids

(cherry picked from commit d58a972)
@924060929 924060929 mentioned this pull request Dec 5, 2024
Merged
924060929 added a commit to 924060929/incubator-doris that referenced this pull request Dec 5, 2024
@924060929
[feature](function) support orthogonal_bitmap_expr_calculate & orthog…
eb8fb92 
…onal_bitmap_expr_calculate_count for nereids (apache#44991)

support orthogonal_bitmap_expr_calculate &
orthogonal_bitmap_expr_calculate_count for nereids

(cherry picked from commit d58a972)
924060929 added a commit that referenced this pull request Dec 5, 2024
@924060929 @starocean999
[feature](function) support orthogonal_bitmap_expr_calculate & orthog…
baedeb2 
…onal_bitmap_expr_calculate_count for nereids (#44671) (#44991) (#45011)

cherry pick from #44671 and #44991

#44671
AlwaysNotNullable agg functions must return a non-null value if it's a scalar agg. So a new base class NotNullableAggregateFunction is introduced, all sub classes must implement resultForEmptyInput method to get the non-null value correctly

#44991
support orthogonal_bitmap_expr_calculate & orthogonal_bitmap_expr_calculate_count for nereids


Co-authored-by: starocean999 <[email protected]>
924060929 added a commit that referenced this pull request Dec 7, 2024
@924060929
[feature](function) support orthogonal_bitmap_expr_calculate & ortho…
2543af8 
…gonal_bitmap_expr_calculate_count for nereids (#44991) (#45012)

support orthogonal_bitmap_expr_calculate & orthogonal_bitmap_expr_calculate_count for nereids
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@morrySnow morrySnow morrySnow approved these changes

@starocean999 starocean999 starocean999 approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.x-conflict dev/2.1.8-merged dev/3.0.4-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@924060929 @doris-robot @starocean999 @morrySnow @wm1581066