Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

branch-3.0: [enhancement](meta) Sync tablet meta even if local state is not running #45821 #45962

Merged
merged 1 commit into from
Dec 26, 2024

Conversation

github-actions[bot]
Copy link
Contributor

Cherry-picked from #45821

…ng (#45821)

Avoid a running tablet keep not running state on single one BE with no
query. Even if it is a tablet with high compaction score, compaction
will fail on this BE since not running state.

Before this PR, scheduled tablet meta sync will skip not running
tablets. In this PR, we include those tablets in meta sync procedure to
avoid long-term inaccurate tablet state.
@Thearas
Copy link
Contributor

Thearas commented Dec 25, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Dec 25, 2024
@Thearas
Copy link
Contributor

Thearas commented Dec 25, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40589 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 84cf0b841654d630244afa03e0abc35ea8741047, data reload: false

------ Round 1 ----------------------------------
q1	17571	7389	7265	7265
q2	2054	172	173	172
q3	11057	1119	1161	1119
q4	10657	767	661	661
q5	7759	2862	2788	2788
q6	234	150	147	147
q7	987	634	612	612
q8	9386	1965	2027	1965
q9	6603	6384	6394	6384
q10	7031	2321	2274	2274
q11	467	268	277	268
q12	399	219	214	214
q13	17802	2964	2999	2964
q14	241	204	204	204
q15	566	511	513	511
q16	696	623	625	623
q17	965	544	653	544
q18	7193	6604	6555	6555
q19	1379	1014	972	972
q20	480	208	206	206
q21	4127	3148	3368	3148
q22	1101	993	1012	993
Total cold run time: 108755 ms
Total hot run time: 40589 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7254	7219	7216	7216
q2	327	240	222	222
q3	2901	2918	2926	2918
q4	2130	1849	1847	1847
q5	5710	5721	5700	5700
q6	226	149	145	145
q7	2270	1856	1842	1842
q8	3338	3528	3431	3431
q9	8967	8843	8786	8786
q10	3604	3589	3550	3550
q11	587	500	496	496
q12	819	678	634	634
q13	9597	3145	3171	3145
q14	320	265	274	265
q15	568	515	518	515
q16	723	684	680	680
q17	1841	1636	1606	1606
q18	8321	7660	7435	7435
q19	1684	1521	1567	1521
q20	2100	1873	1867	1867
q21	5624	5438	5511	5438
q22	1165	1051	1076	1051
Total cold run time: 70076 ms
Total hot run time: 60310 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 197256 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 84cf0b841654d630244afa03e0abc35ea8741047, data reload: false

query1	1230	934	936	934
query2	6258	2079	2036	2036
query3	10949	4328	4295	4295
query4	66429	28830	23503	23503
query5	4939	447	446	446
query6	414	190	180	180
query7	5509	315	315	315
query8	316	227	227	227
query9	8584	2682	2701	2682
query10	460	270	253	253
query11	17244	15179	15742	15179
query12	158	100	106	100
query13	1459	450	444	444
query14	10786	7312	7392	7312
query15	198	177	175	175
query16	6665	491	494	491
query17	1069	615	609	609
query18	1697	314	314	314
query19	223	161	164	161
query20	122	117	124	117
query21	64	45	52	45
query22	4724	4345	4395	4345
query23	34911	34237	34106	34106
query24	6008	2858	2870	2858
query25	550	421	434	421
query26	669	169	170	169
query27	1836	313	312	312
query28	4164	2502	2448	2448
query29	716	483	425	425
query30	240	164	178	164
query31	982	835	816	816
query32	66	53	54	53
query33	440	285	297	285
query34	911	534	527	527
query35	816	775	727	727
query36	1111	982	1006	982
query37	119	73	75	73
query38	4024	4058	4058	4058
query39	1522	1441	1448	1441
query40	142	81	80	80
query41	51	46	46	46
query42	115	103	95	95
query43	526	497	493	493
query44	1185	833	835	833
query45	181	170	169	169
query46	1141	750	722	722
query47	2022	1938	1934	1934
query48	481	385	388	385
query49	719	393	380	380
query50	855	418	416	416
query51	7391	7155	7309	7155
query52	100	83	89	83
query53	263	185	187	185
query54	565	465	459	459
query55	77	79	78	78
query56	253	232	228	228
query57	1256	1099	1099	1099
query58	204	201	201	201
query59	3173	2874	2987	2874
query60	281	256	246	246
query61	107	105	107	105
query62	784	671	661	661
query63	208	185	193	185
query64	1382	697	647	647
query65	3288	3193	3203	3193
query66	719	306	295	295
query67	16017	15556	15615	15556
query68	3933	596	563	563
query69	415	265	256	256
query70	1189	1096	1086	1086
query71	375	252	252	252
query72	6455	4051	4057	4051
query73	753	345	341	341
query74	10123	8993	9072	8993
query75	3357	2594	2647	2594
query76	1820	919	1063	919
query77	478	266	269	266
query78	10659	9652	9630	9630
query79	1206	596	603	596
query80	851	420	432	420
query81	510	242	240	240
query82	1254	119	116	116
query83	164	142	140	140
query84	282	75	72	72
query85	852	293	282	282
query86	338	300	302	300
query87	4456	4269	4394	4269
query88	3500	2397	2341	2341
query89	416	288	282	282
query90	2011	186	183	183
query91	186	148	166	148
query92	63	48	50	48
query93	1237	543	552	543
query94	805	293	293	293
query95	361	257	246	246
query96	610	277	274	274
query97	3294	3193	3244	3193
query98	230	203	200	200
query99	1597	1288	1298	1288
Total cold run time: 315848 ms
Total hot run time: 197256 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.78 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 84cf0b841654d630244afa03e0abc35ea8741047, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.06	0.06
query4	1.63	0.11	0.11
query5	0.54	0.53	0.51
query6	1.13	0.71	0.72
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.57	0.48	0.51
query10	0.56	0.55	0.56
query11	0.14	0.10	0.11
query12	0.14	0.12	0.12
query13	0.61	0.60	0.59
query14	2.92	3.08	3.07
query15	0.90	0.82	0.83
query16	0.36	0.38	0.39
query17	1.10	1.06	1.02
query18	0.24	0.23	0.22
query19	1.89	1.96	1.82
query20	0.01	0.01	0.02
query21	15.40	0.58	0.58
query22	2.85	2.82	1.53
query23	16.91	1.19	0.87
query24	2.61	0.91	1.47
query25	0.29	0.12	0.08
query26	0.32	0.14	0.14
query27	0.06	0.04	0.04
query28	10.66	1.10	1.08
query29	12.54	3.16	3.20
query30	0.25	0.06	0.06
query31	2.86	0.38	0.39
query32	3.26	0.48	0.48
query33	3.03	3.02	3.06
query34	17.08	4.49	4.52
query35	4.51	4.52	4.53
query36	0.66	0.49	0.48
query37	0.10	0.05	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.16	0.13	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.91 s
Total hot run time: 32.78 s

@dataroaring dataroaring merged commit 44cbeac into branch-3.0 Dec 26, 2024
20 of 21 checks passed
@github-actions github-actions bot deleted the auto-pick-45821-branch-3.0 branch December 26, 2024 03:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants