Skip to content

[opt](audit) add many audit info into audit log and table #51363

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

morrySnow
Copy link
Contributor

What problem does this PR solve?

Problem Summary:

add below audit infos

  • parse time
  • plan times
  • get meta times
  • schedule times
  • is sql cache
  • is query cache
  • is internal query
  • spill info
  • variable info

fix audit table could not be alter correctly

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

add below audit infos
- parse time
- plan times
- get meta times
- schedule times
- is sql cache
- is query cache
- is internal query
- spill info
- variable info

fix audit table could not be alter correctly
@Thearas
Copy link
Contributor

Thearas commented May 29, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@morrySnow
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 33914 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 478d37ea1a096758528d7f20fdeafb733b584145, data reload: false

------ Round 1 ----------------------------------
q1	26269	5087	5024	5024
q2	1953	276	198	198
q3	10306	1254	680	680
q4	10264	1003	559	559
q5	8101	2398	2332	2332
q6	183	176	137	137
q7	893	742	610	610
q8	9314	1295	1097	1097
q9	6767	5064	5076	5064
q10	6875	2323	1910	1910
q11	486	288	286	286
q12	349	355	217	217
q13	17770	3733	3150	3150
q14	227	242	220	220
q15	558	492	482	482
q16	423	437	369	369
q17	595	872	355	355
q18	7765	7258	7141	7141
q19	1469	948	561	561
q20	347	335	218	218
q21	4030	3241	2340	2340
q22	1069	1022	964	964
Total cold run time: 116013 ms
Total hot run time: 33914 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5132	5051	5061	5051
q2	239	317	218	218
q3	2160	2650	2323	2323
q4	1332	1794	1319	1319
q5	4592	4432	4361	4361
q6	213	177	130	130
q7	1957	1931	1753	1753
q8	2541	2532	2482	2482
q9	7218	7045	7151	7045
q10	2955	3213	2750	2750
q11	570	506	479	479
q12	677	769	614	614
q13	3615	3893	3338	3338
q14	287	284	260	260
q15	514	484	468	468
q16	435	486	449	449
q17	1176	1713	1402	1402
q18	7640	7595	7581	7581
q19	803	785	887	785
q20	2031	2014	1975	1975
q21	4652	4372	4337	4337
q22	1076	1031	989	989
Total cold run time: 51815 ms
Total hot run time: 50109 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 185112 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 478d37ea1a096758528d7f20fdeafb733b584145, data reload: false

query1	992	495	485	485
query2	6579	1810	1798	1798
query3	6751	216	221	216
query4	25948	23807	22916	22916
query5	4373	649	470	470
query6	298	214	200	200
query7	4638	506	292	292
query8	288	234	225	225
query9	8633	2634	2631	2631
query10	509	315	267	267
query11	15926	15062	14788	14788
query12	167	110	107	107
query13	1643	527	401	401
query14	9812	6259	6188	6188
query15	210	191	165	165
query16	7246	643	488	488
query17	1184	728	570	570
query18	1975	394	299	299
query19	182	180	171	171
query20	120	118	120	118
query21	216	123	108	108
query22	3970	3989	4010	3989
query23	34088	33027	33192	33027
query24	8228	2392	2372	2372
query25	516	464	385	385
query26	1232	269	150	150
query27	2708	486	334	334
query28	4380	2156	2132	2132
query29	788	543	455	455
query30	320	209	194	194
query31	915	853	744	744
query32	73	65	63	63
query33	554	367	318	318
query34	800	855	534	534
query35	801	814	732	732
query36	962	976	895	895
query37	114	100	79	79
query38	4174	4141	4126	4126
query39	1473	1437	1404	1404
query40	205	120	110	110
query41	99	57	58	57
query42	119	106	110	106
query43	487	494	471	471
query44	1289	828	844	828
query45	178	178	174	174
query46	835	1022	626	626
query47	1737	1779	1740	1740
query48	388	422	309	309
query49	745	490	377	377
query50	652	664	406	406
query51	4166	4104	4116	4104
query52	118	105	102	102
query53	215	249	185	185
query54	567	569	495	495
query55	85	80	89	80
query56	321	320	305	305
query57	1130	1131	1069	1069
query58	283	278	270	270
query59	2575	2715	2585	2585
query60	347	323	338	323
query61	150	151	157	151
query62	805	723	685	685
query63	220	189	179	179
query64	4353	1004	677	677
query65	4204	4166	4085	4085
query66	1123	410	310	310
query67	15944	15594	15370	15370
query68	6969	870	527	527
query69	483	308	272	272
query70	1197	1118	1055	1055
query71	420	343	293	293
query72	5536	4771	4973	4771
query73	673	640	352	352
query74	8879	9091	8629	8629
query75	3200	3189	2682	2682
query76	3283	1183	737	737
query77	486	369	296	296
query78	10171	10192	9349	9349
query79	2255	813	572	572
query80	576	534	429	429
query81	503	254	222	222
query82	492	211	97	97
query83	249	251	232	232
query84	254	104	83	83
query85	821	345	327	327
query86	378	309	295	295
query87	4343	4522	4375	4375
query88	3980	2305	2310	2305
query89	387	316	295	295
query90	1941	214	212	212
query91	144	141	113	113
query92	76	60	58	58
query93	1873	946	593	593
query94	694	414	290	290
query95	380	303	287	287
query96	491	575	289	289
query97	2742	2758	2598	2598
query98	235	204	208	204
query99	1323	1418	1258	1258
Total cold run time: 272560 ms
Total hot run time: 185112 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.69 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 478d37ea1a096758528d7f20fdeafb733b584145, data reload: false

query1	0.05	0.03	0.03
query2	0.12	0.10	0.11
query3	0.26	0.20	0.20
query4	1.59	0.20	0.11
query5	0.44	0.42	0.42
query6	1.19	0.67	0.66
query7	0.02	0.02	0.02
query8	0.05	0.03	0.04
query9	0.59	0.53	0.52
query10	0.58	0.57	0.57
query11	0.15	0.11	0.11
query12	0.15	0.11	0.12
query13	0.62	0.60	0.59
query14	0.78	0.80	0.80
query15	0.88	0.85	0.85
query16	0.38	0.38	0.38
query17	1.03	1.05	1.00
query18	0.22	0.21	0.20
query19	1.93	1.80	1.80
query20	0.01	0.01	0.01
query21	15.47	0.91	0.54
query22	0.76	1.18	0.94
query23	14.74	1.38	0.62
query24	6.65	1.88	1.35
query25	0.51	0.25	0.06
query26	0.52	0.16	0.15
query27	0.06	0.05	0.04
query28	10.06	0.79	0.44
query29	12.73	3.96	3.28
query30	0.25	0.09	0.07
query31	2.84	0.61	0.39
query32	3.24	0.54	0.48
query33	3.02	3.06	3.11
query34	15.85	5.11	4.45
query35	4.48	4.52	4.50
query36	0.67	0.51	0.49
query37	0.09	0.07	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.18	0.14	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 103.39 s
Total hot run time: 29.69 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants