Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[opt](join) Check the property of nullable from intermediate row #45017

Merged
merged 1 commit into from
Dec 16, 2024
Merged

[opt](join) Check the property of nullable from intermediate row #45017

merged 1 commit into from
Dec 16, 2024

Conversation

mrhhsg
Copy link
Member

@mrhhsg mrhhsg commented Dec 5, 2024

What problem does this PR solve?

In actual use cases, there have been cases where the nullable property passed from the FE is incorrect, so it is necessary to add checks in the BE to prevent crashes.

Related PR: #38556

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@mrhhsg
Copy link
Member Author

mrhhsg commented Dec 5, 2024

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 38.47% (10008/26013)
Line Coverage: 29.51% (83889/284257)
Region Coverage: 28.63% (43129/150668)
Branch Coverage: 25.22% (21921/86926)
Coverage Report: http://coverage.selectdb-in.cc/coverage/40a8ee4d622f9bef4de977ba3179196f103b40ec_40a8ee4d622f9bef4de977ba3179196f103b40ec/report/index.html

@mrhhsg
Copy link
Member Author

mrhhsg commented Dec 5, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 36979 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit cdec4eb98f2a0a4a323c69d17de578ed8f740d24, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17583	7497	7265	7265
q2	2079	184	168	168
q3	10706	1066	1174	1066
q4	10567	746	697	697
q5	7600	2724	2715	2715
q6	235	145	146	145
q7	1016	655	585	585
q8	9234	1885	1949	1885
q9	6686	6508	6502	6502
q10	7018	2285	2303	2285
q11	465	263	261	261
q12	431	211	213	211
q13	q14	239	231	214	214
q15	584	546	520	520
q16	648	596	585	585
q17	966	572	516	516
q18	7286	6724	6668	6668
q19	1224	1031	985	985
q20	395	191	179	179
q21	4212	3216	3219	3216
q22	370	313	311	311
Total cold run time: 89544 ms
Total hot run time: 36979 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	7302	7208	7226	7208
q2	329	234	223	223
q3	2921	2695	2806	2695
q4	2041	1822	1745	1745
q5	5317	5398	5457	5398
q6	224	140	140	140
q7	2256	1776	1837	1776
q8	3383	3595	3500	3500
q9	9057	9073	9109	9073
q10	3585	3622	3517	3517
q11	583	489	500	489
q12	832	628	659	628
q13	q14	314	284	297	284
q15	588	522	527	522
q16	649	626	654	626
q17	1884	1627	1598	1598
q18	8247	7790	7731	7731
q19	1695	1540	1448	1448
q20	2119	1901	1919	1901
q21	5679	5587	5436	5436
q22	636	563	565	563
Total cold run time: 59641 ms
Total hot run time: 56501 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 197683 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit cdec4eb98f2a0a4a323c69d17de578ed8f740d24, data reload: false

query1	1531	932	900	900
query2	6243	2033	2032	2032
query3	10965	4472	4512	4472
query4	62877	29523	23538	23538
query5	5217	457	448	448
query6	395	180	176	176
query7	5454	306	304	304
query8	323	237	244	237
query9	8598	2737	2717	2717
query10	437	248	237	237
query11	17597	15180	15842	15180
query12	155	106	108	106
query13	1470	439	426	426
query14	10627	7145	7417	7145
query15	212	186	184	184
query16	7162	471	458	458
query17	1241	566	560	560
query18	1831	304	313	304
query19	198	151	146	146
query20	118	118	121	118
query21	213	105	106	105
query22	4887	4654	4756	4654
query23	35091	34555	34714	34555
query24	5552	2519	2482	2482
query25	509	401	413	401
query26	643	154	152	152
query27	1818	289	293	289
query28	4412	2532	2526	2526
query29	697	419	401	401
query30	205	148	155	148
query31	1058	853	865	853
query32	62	52	52	52
query33	422	285	273	273
query34	938	505	515	505
query35	867	783	753	753
query36	1089	980	969	969
query37	124	77	81	77
query38	4477	4424	4357	4357
query39	1520	1486	1490	1486
query40	199	101	97	97
query41	43	43	45	43
query42	115	96	94	94
query43	543	483	489	483
query44	1193	818	827	818
query45	187	174	170	170
query46	1170	721	723	721
query47	2050	1979	1971	1971
query48	421	319	319	319
query49	739	387	425	387
query50	824	402	390	390
query51	7274	7274	7365	7274
query52	105	92	92	92
query53	261	180	191	180
query54	540	389	393	389
query55	80	79	75	75
query56	299	245	267	245
query57	1329	1175	1166	1166
query58	222	213	205	205
query59	3248	3031	3174	3031
query60	283	243	234	234
query61	108	106	100	100
query62	771	660	656	656
query63	209	184	180	180
query64	1415	679	634	634
query65	3274	3201	3210	3201
query66	714	301	298	298
query67	16093	16047	15664	15664
query68	4149	561	539	539
query69	423	250	257	250
query70	1217	1133	1157	1133
query71	368	248	256	248
query72	6383	4033	4096	4033
query73	768	369	374	369
query74	10120	9046	8919	8919
query75	3418	2630	2675	2630
query76	2020	1047	1152	1047
query77	474	279	264	264
query78	10606	9499	9416	9416
query79	1562	600	596	596
query80	884	477	424	424
query81	506	224	232	224
query82	1254	112	117	112
query83	264	152	143	143
query84	284	72	69	69
query85	888	295	290	290
query86	326	303	300	300
query87	4602	4520	4467	4467
query88	3509	2224	2219	2219
query89	412	299	298	298
query90	2030	184	184	184
query91	143	104	103	103
query92	64	54	49	49
query93	1978	542	557	542
query94	728	263	297	263
query95	343	249	245	245
query96	617	272	286	272
query97	2858	2707	2713	2707
query98	223	199	197	197
query99	1626	1288	1308	1288
Total cold run time: 315991 ms
Total hot run time: 197683 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.65 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit cdec4eb98f2a0a4a323c69d17de578ed8f740d24, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.04	0.04
query3	0.24	0.07	0.07
query4	1.62	0.11	0.10
query5	0.43	0.41	0.41
query6	1.14	0.66	0.65
query7	0.02	0.02	0.01
query8	0.04	0.03	0.04
query9	0.57	0.51	0.50
query10	0.56	0.56	0.56
query11	0.13	0.11	0.11
query12	0.14	0.11	0.11
query13	0.61	0.60	0.61
query14	2.71	2.74	2.72
query15	0.90	0.83	0.83
query16	0.37	0.39	0.38
query17	1.03	1.07	1.06
query18	0.21	0.21	0.21
query19	1.93	1.81	2.01
query20	0.01	0.01	0.01
query21	15.36	0.59	0.58
query22	2.79	2.20	1.50
query23	17.16	0.89	0.77
query24	2.61	1.35	1.50
query25	0.20	0.20	0.05
query26	0.52	0.13	0.14
query27	0.05	0.05	0.04
query28	10.45	1.10	1.07
query29	12.54	3.22	3.23
query30	0.25	0.06	0.06
query31	2.86	0.39	0.38
query32	3.24	0.47	0.46
query33	2.99	3.06	3.06
query34	16.88	4.45	4.45
query35	4.58	4.50	4.50
query36	0.67	0.47	0.48
query37	0.09	0.06	0.06
query38	0.04	0.04	0.04
query39	0.03	0.03	0.03
query40	0.17	0.14	0.12
query41	0.07	0.02	0.02
query42	0.03	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.38 s
Total hot run time: 32.65 s

@mrhhsg
Copy link
Member Author

mrhhsg commented Dec 5, 2024

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 38.48% (10006/26003)
Line Coverage: 29.51% (83913/284315)
Region Coverage: 28.61% (43123/150707)
Branch Coverage: 25.20% (21917/86966)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c948461d5605f2393740f7c3f72b5b5bdf00fa26_c948461d5605f2393740f7c3f72b5b5bdf00fa26/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 39943 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c948461d5605f2393740f7c3f72b5b5bdf00fa26, data reload: false

------ Round 1 ----------------------------------
q1	17583	7474	7309	7309
q2	2046	188	172	172
q3	10629	1125	1112	1112
q4	10557	724	670	670
q5	7612	2714	2712	2712
q6	239	150	148	148
q7	994	633	603	603
q8	9254	1814	1881	1814
q9	6654	6448	6490	6448
q10	7026	2317	2383	2317
q11	475	261	261	261
q12	425	222	216	216
q13	17774	2994	3049	2994
q14	241	209	209	209
q15	560	530	525	525
q16	681	580	600	580
q17	969	573	582	573
q18	7382	6932	6602	6602
q19	1339	1048	1053	1048
q20	464	184	183	183
q21	4137	3129	3135	3129
q22	388	318	321	318
Total cold run time: 107429 ms
Total hot run time: 39943 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7224	7231	7231	7231
q2	326	226	229	226
q3	2920	2786	2955	2786
q4	2076	1779	1818	1779
q5	5688	5660	5655	5655
q6	229	145	138	138
q7	2251	1818	1815	1815
q8	3447	3550	3490	3490
q9	8959	9070	9030	9030
q10	3579	3562	3575	3562
q11	602	508	515	508
q12	837	586	617	586
q13	12024	3219	3233	3219
q14	303	262	277	262
q15	600	542	530	530
q16	701	654	651	651
q17	1854	1663	1595	1595
q18	8380	7769	7419	7419
q19	1731	1634	1640	1634
q20	2093	1856	1867	1856
q21	5664	5482	5520	5482
q22	665	582	578	578
Total cold run time: 72153 ms
Total hot run time: 60032 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 198301 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c948461d5605f2393740f7c3f72b5b5bdf00fa26, data reload: false

query1	1466	974	968	968
query2	6243	2071	2032	2032
query3	10973	4468	4517	4468
query4	67023	28898	23683	23683
query5	4953	471	455	455
query6	406	182	185	182
query7	5487	307	308	307
query8	329	250	237	237
query9	8747	2749	2714	2714
query10	430	257	249	249
query11	17274	15360	15832	15360
query12	159	106	110	106
query13	1433	409	410	409
query14	10655	8055	7387	7387
query15	219	189	201	189
query16	7204	459	470	459
query17	1310	606	606	606
query18	1759	332	320	320
query19	226	159	165	159
query20	119	119	132	119
query21	214	115	107	107
query22	4567	4619	4428	4428
query23	34999	34733	34758	34733
query24	5546	2697	2457	2457
query25	483	377	411	377
query26	655	152	159	152
query27	1844	288	298	288
query28	4379	2542	2460	2460
query29	703	435	452	435
query30	211	156	150	150
query31	1035	804	856	804
query32	71	57	63	57
query33	414	290	296	290
query34	930	518	531	518
query35	920	782	769	769
query36	1099	953	981	953
query37	125	77	74	74
query38	4496	4399	4440	4399
query39	1532	1510	1478	1478
query40	222	103	98	98
query41	46	42	43	42
query42	124	106	100	100
query43	544	506	501	501
query44	1232	811	837	811
query45	189	185	178	178
query46	1201	744	752	744
query47	2008	1910	1929	1910
query48	423	309	324	309
query49	731	387	397	387
query50	903	403	409	403
query51	7378	7207	7034	7034
query52	103	89	88	88
query53	261	191	181	181
query54	525	412	395	395
query55	81	74	76	74
query56	260	263	242	242
query57	1278	1112	1116	1112
query58	217	223	220	220
query59	3183	2998	3038	2998
query60	272	260	270	260
query61	114	134	108	108
query62	784	666	687	666
query63	214	192	190	190
query64	1368	697	687	687
query65	3350	3182	3222	3182
query66	697	297	306	297
query67	15933	15714	15646	15646
query68	4262	572	572	572
query69	433	261	264	261
query70	1208	1126	1163	1126
query71	384	255	255	255
query72	6435	4116	4078	4078
query73	768	367	379	367
query74	10338	8982	8968	8968
query75	3438	2682	2751	2682
query76	1719	965	1083	965
query77	484	277	273	273
query78	10611	9472	9411	9411
query79	1434	622	604	604
query80	872	446	439	439
query81	484	288	227	227
query82	1294	127	124	124
query83	184	150	145	145
query84	284	71	67	67
query85	859	301	313	301
query86	348	298	302	298
query87	4705	4662	4552	4552
query88	3259	2254	2206	2206
query89	418	294	300	294
query90	2011	189	191	189
query91	144	103	113	103
query92	70	50	52	50
query93	1705	544	548	544
query94	824	302	304	302
query95	359	248	249	248
query96	629	272	277	272
query97	2851	2699	2672	2672
query98	218	199	197	197
query99	1590	1308	1319	1308
Total cold run time: 318901 ms
Total hot run time: 198301 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.91 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c948461d5605f2393740f7c3f72b5b5bdf00fa26, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.03	0.03
query3	0.23	0.08	0.06
query4	1.62	0.10	0.11
query5	0.43	0.41	0.42
query6	1.19	0.65	0.66
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.60	0.51	0.51
query10	0.56	0.55	0.58
query11	0.15	0.10	0.10
query12	0.14	0.12	0.11
query13	0.62	0.62	0.59
query14	2.82	2.84	2.83
query15	0.91	0.83	0.83
query16	0.38	0.40	0.38
query17	1.05	1.08	1.02
query18	0.22	0.21	0.20
query19	1.94	1.86	2.00
query20	0.02	0.01	0.01
query21	15.36	0.57	0.58
query22	2.69	1.80	1.84
query23	16.96	1.05	0.76
query24	3.32	1.99	1.31
query25	0.26	0.10	0.19
query26	0.51	0.15	0.14
query27	0.05	0.05	0.05
query28	9.55	1.11	1.08
query29	12.55	3.22	3.25
query30	0.25	0.06	0.06
query31	2.84	0.38	0.38
query32	3.29	0.46	0.44
query33	2.99	3.02	3.04
query34	16.86	4.45	4.40
query35	4.50	4.47	4.48
query36	0.64	0.48	0.50
query37	0.09	0.06	0.06
query38	0.04	0.04	0.03
query39	0.03	0.03	0.03
query40	0.18	0.13	0.13
query41	0.07	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.04	0.03
Total cold run time: 106.12 s
Total hot run time: 32.91 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 16, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

HappenLee
HappenLee approved these changes Dec 16, 2024
View reviewed changes
Copy link
Contributor

@HappenLee HappenLee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@yiguolei yiguolei merged commit 79eddb7 into apache:master Dec 16, 2024
27 of 29 checks passed
github-actions bot pushed a commit that referenced this pull request Dec 16, 2024
@mrhhsg
[opt](join) Check the property of nullable from intermediate row (#45017
349cf5a 
)
@github-actions github-actions bot mentioned this pull request Dec 16, 2024
Open
github-actions bot pushed a commit that referenced this pull request Dec 16, 2024
@mrhhsg
[opt](join) Check the property of nullable from intermediate row (#45017
f6958fa 
)
@github-actions github-actions bot mentioned this pull request Dec 16, 2024
Merged
yiguolei pushed a commit that referenced this pull request Dec 18, 2024
@github-actions @mrhhsg
branch-2.1: [opt](join) Check the property of nullable from intermedi…
59b3760 
…ate row #45017 (#45476)

Cherry-picked from #45017

Co-authored-by: Jerry Hu <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@HappenLee HappenLee HappenLee approved these changes

@yiguolei yiguolei yiguolei approved these changes

@BiteTheDDDDt BiteTheDDDDt BiteTheDDDDt approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.8-merged dev/3.0.x reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@mrhhsg @doris-robot @BiteTheDDDDt @yiguolei @HappenLee