Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[enhance](mtmv)MTMV no longer generate cache when replaying logs #44283

Merged
merged 2 commits into from
Nov 20, 2024
Merged

[enhance](mtmv)MTMV no longer generate cache when replaying logs #44283

merged 2 commits into from
Nov 20, 2024

Conversation

zddr
Copy link
Contributor

@zddr zddr commented Nov 19, 2024 •
edited
Loading

What problem does this PR solve?

When replaying logs, cache will no longer be generated because the catalog may not have been initialized or the dependent environment may not be connected, causing it to freeze here and preventing FE from starting

The cost is that after the materialized view is refreshed, the cache of the follower node will be empty, and a cache will be generated when the query is first used

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:
MTMV no longer generate cache when replaying logs

Release note

MTMV no longer generate cache when replaying logs

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@zddr
Copy link
Contributor Author

zddr commented Nov 19, 2024

run buildall

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 19, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 45241 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ba0338f77364874ccf02813683652f52c488943a, data reload: false

------ Round 1 ----------------------------------
q1	17624	7514	7315	7315
q2	2244	1170	1185	1170
q3	10075	1191	1207	1191
q4	10304	747	767	747
q5	7600	2809	2756	2756
q6	236	151	147	147
q7	999	633	613	613
q8	9406	2361	2383	2361
q9	6575	6476	6429	6429
q10	7029	2349	2316	2316
q11	472	260	259	259
q12	418	214	213	213
q13	17811	3015	3036	3015
q14	246	216	212	212
q15	584	541	513	513
q16	655	595	580	580
q17	984	539	616	539
q18	7455	6837	6664	6664
q19	1335	1081	972	972
q20	2927	2682	2705	2682
q21	3965	3205	3319	3205
q22	1394	1342	1344	1342
Total cold run time: 110338 ms
Total hot run time: 45241 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7351	7310	7429	7310
q2	337	229	232	229
q3	3077	3071	2994	2994
q4	2072	1879	1785	1785
q5	5643	5727	5712	5712
q6	228	143	136	136
q7	2236	1839	1825	1825
q8	3281	3565	3521	3521
q9	8972	8883	8926	8883
q10	3608	3612	3560	3560
q11	607	518	511	511
q12	838	632	632	632
q13	10913	3217	3322	3217
q14	311	274	272	272
q15	588	536	514	514
q16	680	644	651	644
q17	1868	1656	1608	1608
q18	8407	7704	7848	7704
q19	1695	1629	1461	1461
q20	2137	1899	1898	1898
q21	5537	5385	5427	5385
q22	629	586	576	576
Total cold run time: 71015 ms
Total hot run time: 60377 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.84 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ba0338f77364874ccf02813683652f52c488943a, data reload: false

query1	0.03	0.04	0.03
query2	0.07	0.03	0.04
query3	0.24	0.07	0.07
query4	1.63	0.10	0.11
query5	0.42	0.43	0.40
query6	1.13	0.65	0.66
query7	0.02	0.02	0.02
query8	0.05	0.04	0.03
query9	0.58	0.50	0.49
query10	0.55	0.55	0.57
query11	0.14	0.11	0.10
query12	0.14	0.11	0.11
query13	0.62	0.61	0.59
query14	2.73	2.82	2.74
query15	0.91	0.83	0.84
query16	0.40	0.41	0.38
query17	0.99	1.00	1.08
query18	0.20	0.19	0.20
query19	1.91	1.79	2.08
query20	0.01	0.01	0.02
query21	15.36	0.60	0.58
query22	2.76	1.70	2.50
query23	16.88	0.93	0.67
query24	3.01	0.66	1.00
query25	0.14	0.05	0.08
query26	0.54	0.15	0.13
query27	0.04	0.05	0.04
query28	10.97	1.10	1.07
query29	12.58	3.26	3.26
query30	0.25	0.07	0.06
query31	2.85	0.37	0.38
query32	3.26	0.47	0.48
query33	3.01	2.95	3.08
query34	17.14	4.43	4.49
query35	4.55	4.53	4.50
query36	0.67	0.50	0.47
query37	0.09	0.06	0.06
query38	0.04	0.04	0.03
query39	0.03	0.03	0.02
query40	0.17	0.14	0.13
query41	0.08	0.03	0.03
query42	0.04	0.02	0.02
query43	0.04	0.03	0.04
Total cold run time: 107.27 s
Total hot run time: 31.84 s

@zddr
Copy link
Contributor Author

zddr commented Nov 20, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 44999 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8ed0d3922514c989aa566bc7374832aabb5bad70, data reload: false

------ Round 1 ----------------------------------
q1	17593	7479	7243	7243
q2	2231	1170	1183	1170
q3	9971	1154	1227	1154
q4	10216	732	703	703
q5	7557	2681	2716	2681
q6	241	152	146	146
q7	999	622	601	601
q8	9376	2364	2368	2364
q9	6622	6392	6344	6344
q10	7044	2282	2311	2282
q11	461	268	257	257
q12	422	219	222	219
q13	17783	3052	3053	3052
q14	241	215	215	215
q15	565	535	529	529
q16	659	596	601	596
q17	974	587	591	587
q18	7262	6769	6717	6717
q19	1336	979	1033	979
q20	2921	2708	2693	2693
q21	3967	3311	3117	3117
q22	1393	1350	1353	1350
Total cold run time: 109834 ms
Total hot run time: 44999 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7287	7237	7406	7237
q2	335	232	236	232
q3	3066	2973	3047	2973
q4	2064	1809	1809	1809
q5	5622	5646	5702	5646
q6	222	147	147	147
q7	2212	1802	1832	1802
q8	3334	3517	3513	3513
q9	8882	8849	8850	8849
q10	3598	3584	3569	3569
q11	585	526	510	510
q12	864	643	632	632
q13	10627	3254	3325	3254
q14	312	290	265	265
q15	576	531	514	514
q16	682	659	648	648
q17	1879	1624	1626	1624
q18	8323	7722	7869	7722
q19	1685	1568	1529	1529
q20	2101	1870	1913	1870
q21	5611	5487	5396	5396
q22	627	571	562	562
Total cold run time: 70494 ms
Total hot run time: 60303 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.1 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8ed0d3922514c989aa566bc7374832aabb5bad70, data reload: false

query1	0.03	0.04	0.03
query2	0.07	0.03	0.03
query3	0.23	0.07	0.06
query4	1.62	0.10	0.10
query5	0.42	0.41	0.40
query6	1.13	0.65	0.65
query7	0.02	0.01	0.02
query8	0.04	0.04	0.02
query9	0.57	0.50	0.50
query10	0.56	0.55	0.56
query11	0.15	0.11	0.11
query12	0.14	0.11	0.11
query13	0.61	0.61	0.60
query14	2.77	2.79	2.83
query15	0.93	0.84	0.84
query16	0.38	0.37	0.38
query17	1.00	1.04	1.04
query18	0.21	0.20	0.20
query19	2.00	2.02	1.91
query20	0.02	0.01	0.01
query21	15.37	0.58	0.60
query22	2.49	1.68	1.98
query23	17.31	0.88	0.76
query24	2.41	0.62	2.01
query25	0.29	0.14	0.05
query26	0.51	0.14	0.14
query27	0.04	0.04	0.04
query28	10.64	1.10	1.07
query29	12.58	3.23	3.24
query30	0.24	0.06	0.06
query31	2.86	0.38	0.39
query32	3.26	0.48	0.49
query33	2.99	2.99	3.09
query34	16.90	4.50	4.43
query35	4.56	4.50	4.47
query36	0.67	0.49	0.48
query37	0.10	0.06	0.06
query38	0.04	0.04	0.03
query39	0.03	0.02	0.02
query40	0.17	0.13	0.13
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.03	0.04	0.03
Total cold run time: 106.5 s
Total hot run time: 32.1 s

@starocean999 starocean999 merged commit 946b4ae into apache:master Nov 20, 2024
25 of 26 checks passed
github-actions bot pushed a commit that referenced this pull request Nov 20, 2024
@zddr
[enhance](mtmv)MTMV no longer generate cache when replaying logs (#44283
55a1534 
)

### What problem does this PR solve?

When replaying logs, cache will no longer be generated because the
catalog may not have been initialized or the dependent environment may
not be connected, causing it to freeze here and preventing FE from
starting

The cost is that after the materialized view is refreshed, the cache of
the follower node will be empty, and a cache will be generated when the
query is first used

Problem Summary:
MTMV no longer generate cache when replaying logs
@github-actions github-actions bot mentioned this pull request Nov 20, 2024
Merged
github-actions bot pushed a commit that referenced this pull request Nov 20, 2024
@zddr
[enhance](mtmv)MTMV no longer generate cache when replaying logs (#44283
d8adde3 
)

### What problem does this PR solve?

When replaying logs, cache will no longer be generated because the
catalog may not have been initialized or the dependent environment may
not be connected, causing it to freeze here and preventing FE from
starting

The cost is that after the materialized view is refreshed, the cache of
the follower node will be empty, and a cache will be generated when the
query is first used

Problem Summary:
MTMV no longer generate cache when replaying logs
@github-actions github-actions bot mentioned this pull request Nov 20, 2024
Merged
yiguolei pushed a commit that referenced this pull request Nov 25, 2024
@github-actions @zddr
branch-2.1: [enhance](mtmv)MTMV no longer generate cache when replayi…
9e0ece2 
…ng logs #44283 (#44308)

Cherry-picked from #44283

Co-authored-by: zhangdong <[email protected]>
dataroaring pushed a commit that referenced this pull request Nov 26, 2024
@github-actions @zddr
branch-3.0: [enhance](mtmv)MTMV no longer generate cache when replayi…
1b06f02 
…ng logs #44283 (#44307)

Cherry-picked from #44283

Co-authored-by: zhangdong <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@morrySnow morrySnow morrySnow approved these changes

@starocean999 starocean999 starocean999 approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.8-merged dev/3.0.4-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@zddr @doris-robot @starocean999 @morrySnow @yiguolei @dataroaring