Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[enhance](mtmv)Mv refresh on commit #34548

Merged
merged 38 commits into from
May 29, 2024
Merged

[enhance](mtmv)Mv refresh on commit #34548

merged 38 commits into from
May 29, 2024

Conversation

zddr
Copy link
Contributor

@zddr zddr commented May 8, 2024 •
edited
Loading

Proposed changes

Issue Number: close #xxx

support refresh MTMV when base table data change / drop partition/replace partition

CREATE MATERIALIZED VIEW mv1
        REFRESH ON COMMIT
        AS 
        SELECT xxx;

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...
@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.
@zddr zddr marked this pull request as draft May 8, 2024 10:57
@zddr zddr changed the title [enhance](mtmv)Mv refresh by base May 9, 2024
@zddr zddr marked this pull request as ready for review May 11, 2024 08:40
@doris-robot
Copy link

TPC-H: Total hot run time: 41853 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 93a834d3386d065ebfa8ad56185bd527bbf83ea7, data reload: false

------ Round 1 ----------------------------------
q1	17617	4246	4197	4197
q2	2021	193	193	193
q3	10466	1203	1161	1161
q4	10187	872	826	826
q5	7469	2730	2689	2689
q6	217	129	136	129
q7	991	619	636	619
q8	9229	2123	2102	2102
q9	9515	6696	6653	6653
q10	9296	3960	3900	3900
q11	442	244	232	232
q12	421	230	218	218
q13	17300	3178	3274	3178
q14	283	249	238	238
q15	513	475	477	475
q16	497	386	397	386
q17	998	744	731	731
q18	8305	7823	7766	7766
q19	6876	1574	1489	1489
q20	642	326	313	313
q21	5131	4107	4080	4080
q22	364	278	282	278
Total cold run time: 118780 ms
Total hot run time: 41853 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4599	4379	4357	4357
q2	372	284	281	281
q3	3163	2927	2952	2927
q4	1907	1656	1622	1622
q5	5476	5478	5571	5478
q6	220	131	134	131
q7	2239	1868	1861	1861
q8	3342	3491	3460	3460
q9	8787	8824	8819	8819
q10	4028	3799	3904	3799
q11	609	526	496	496
q12	829	639	659	639
q13	17178	3241	3254	3241
q14	330	276	293	276
q15	544	492	494	492
q16	522	452	456	452
q17	1862	1549	1518	1518
q18	7790	7998	7673	7673
q19	1690	1589	1555	1555
q20	2063	1862	1876	1862
q21	9770	5090	5011	5011
q22	635	525	524	524
Total cold run time: 77955 ms
Total hot run time: 56474 ms
@doris-robot
Copy link

TPC-DS: Total hot run time: 169467 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 93a834d3386d065ebfa8ad56185bd527bbf83ea7, data reload: false

query1	1846	393	389	389
query2	7843	2490	2357	2357
query3	6641	210	208	208
query4	19477	17293	17221	17221
query5	4112	428	418	418
query6	247	155	168	155
query7	4585	298	293	293
query8	244	182	188	182
query9	8400	2372	2338	2338
query10	450	288	271	271
query11	10533	9979	10012	9979
query12	141	89	93	89
query13	1630	369	363	363
query14	10270	7165	6842	6842
query15	213	168	172	168
query16	7799	274	259	259
query17	1706	527	507	507
query18	1957	272	281	272
query19	217	157	156	156
query20	97	87	87	87
query21	199	156	134	134
query22	4249	4050	3904	3904
query23	33694	33285	33099	33099
query24	11371	2894	2939	2894
query25	645	384	354	354
query26	1754	161	158	158
query27	2898	322	314	314
query28	7501	2047	2020	2020
query29	1078	603	604	603
query30	226	148	150	148
query31	961	747	750	747
query32	95	53	52	52
query33	758	276	278	276
query34	978	483	491	483
query35	761	642	604	604
query36	1071	911	929	911
query37	178	62	66	62
query38	2936	2784	2780	2780
query39	842	772	779	772
query40	265	124	127	124
query41	47	45	45	45
query42	102	98	100	98
query43	581	543	594	543
query44	1251	747	750	747
query45	183	164	165	164
query46	1085	732	708	708
query47	1848	1745	1754	1745
query48	381	307	303	303
query49	967	377	398	377
query50	772	400	390	390
query51	6869	6803	6788	6788
query52	101	87	94	87
query53	356	307	292	292
query54	959	434	429	429
query55	74	73	72	72
query56	267	244	242	242
query57	1143	1054	1072	1054
query58	222	222	214	214
query59	3488	3302	3166	3166
query60	287	253	253	253
query61	88	89	86	86
query62	605	455	447	447
query63	309	291	305	291
query64	9767	2190	1698	1698
query65	3221	3119	3126	3119
query66	1360	321	321	321
query67	15222	14724	14840	14724
query68	4529	522	533	522
query69	470	281	270	270
query70	1077	1095	1181	1095
query71	417	278	272	272
query72	7826	2713	2544	2544
query73	715	325	336	325
query74	5969	5564	5624	5564
query75	3599	2615	2622	2615
query76	3140	993	1008	993
query77	538	269	275	269
query78	10131	10015	9833	9833
query79	2106	527	506	506
query80	907	443	446	443
query81	513	226	222	222
query82	585	91	90	90
query83	242	169	168	168
query84	245	85	88	85
query85	1749	277	367	277
query86	473	318	305	305
query87	3295	3120	3100	3100
query88	4126	2442	2461	2442
query89	473	402	411	402
query90	2064	192	192	192
query91	125	99	96	96
query92	66	50	48	48
query93	1998	512	496	496
query94	1260	190	188	188
query95	405	312	316	312
query96	594	279	277	277
query97	3173	3031	3029	3029
query98	243	223	228	223
query99	1199	868	870	868
Total cold run time: 276675 ms
Total hot run time: 169467 ms
@doris-robot
Copy link

ClickBench: Total hot run time: 29.68 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 93a834d3386d065ebfa8ad56185bd527bbf83ea7, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.06
query4	1.67	0.08	0.08
query5	0.50	0.50	0.49
query6	1.11	0.72	0.72
query7	0.02	0.02	0.02
query8	0.05	0.05	0.04
query9	0.53	0.48	0.48
query10	0.54	0.56	0.53
query11	0.16	0.11	0.11
query12	0.16	0.12	0.12
query13	0.59	0.59	0.58
query14	0.76	0.78	0.78
query15	0.82	0.81	0.80
query16	0.37	0.37	0.37
query17	0.98	1.02	1.02
query18	0.22	0.23	0.25
query19	1.86	1.73	1.82
query20	0.01	0.01	0.01
query21	15.45	0.70	0.68
query22	4.65	7.89	1.02
query23	18.29	1.30	1.27
query24	1.62	0.33	0.24
query25	0.14	0.08	0.08
query26	0.26	0.17	0.17
query27	0.08	0.07	0.08
query28	13.27	1.00	0.98
query29	12.66	3.34	3.27
query30	0.24	0.05	0.05
query31	2.88	0.38	0.38
query32	3.30	0.47	0.46
query33	2.87	2.89	2.87
query34	17.16	4.40	4.47
query35	4.52	4.45	4.54
query36	0.74	0.49	0.49
query37	0.18	0.16	0.15
query38	0.16	0.14	0.15
query39	0.04	0.03	0.04
query40	0.16	0.15	0.13
query41	0.08	0.04	0.04
query42	0.06	0.04	0.04
query43	0.04	0.04	0.04
Total cold run time: 109.55 s
Total hot run time: 29.68 s
mymeiyi
mymeiyi approved these changes May 27, 2024
View reviewed changes
Copy link
Contributor

@mymeiyi mymeiyi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM
@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label May 27, 2024
@zddr
Copy link
Contributor Author

zddr commented May 27, 2024

run p0
@morrySnow morrySnow marked this pull request as draft May 28, 2024 04:14
@zddr zddr marked this pull request as ready for review May 28, 2024 11:42
@zddr
Copy link
Contributor Author

zddr commented May 28, 2024

run buildall
@zddr
Copy link
Contributor Author

zddr commented May 28, 2024

run buildall
@doris-robot
Copy link

TPC-H: Total hot run time: 40286 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 143776a229fd384a79ac4b99d45b04f6dfa2ecf7, data reload: false

------ Round 1 ----------------------------------
q1	18220	4449	4398	4398
q2	2569	205	190	190
q3	11414	1231	1270	1231
q4	10485	822	775	775
q5	7590	2747	2841	2747
q6	217	134	133	133
q7	965	609	606	606
q8	9384	2108	2081	2081
q9	8893	6548	6487	6487
q10	8925	3725	3699	3699
q11	458	248	243	243
q12	442	220	213	213
q13	17760	3017	3017	3017
q14	277	216	232	216
q15	535	477	478	477
q16	512	398	382	382
q17	969	660	718	660
q18	8046	7463	7322	7322
q19	3194	1553	1523	1523
q20	641	322	317	317
q21	5025	3288	3899	3288
q22	345	284	281	281
Total cold run time: 116866 ms
Total hot run time: 40286 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4324	4191	4188	4188
q2	377	266	271	266
q3	3013	2773	2729	2729
q4	1866	1593	1580	1580
q5	5234	5321	5282	5282
q6	217	124	127	124
q7	2131	1778	1791	1778
q8	3209	3350	3286	3286
q9	8354	8439	8364	8364
q10	3864	3720	3663	3663
q11	585	499	498	498
q12	739	606	590	590
q13	17431	3015	2985	2985
q14	286	272	259	259
q15	507	474	477	474
q16	492	425	437	425
q17	1762	1498	1466	1466
q18	7671	7563	7277	7277
q19	2343	1644	1534	1534
q20	2001	1791	1802	1791
q21	4917	4645	4600	4600
q22	602	530	514	514
Total cold run time: 71925 ms
Total hot run time: 53673 ms
@doris-robot
Copy link

TPC-DS: Total hot run time: 169541 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 143776a229fd384a79ac4b99d45b04f6dfa2ecf7, data reload: false

query1	939	393	381	381
query2	6467	2616	2342	2342
query3	6650	200	206	200
query4	20114	17436	17267	17267
query5	4122	414	417	414
query6	240	159	151	151
query7	4595	306	291	291
query8	244	183	177	177
query9	8608	2457	2445	2445
query10	453	280	265	265
query11	10588	10062	10326	10062
query12	135	90	88	88
query13	1641	384	371	371
query14	10397	6825	7644	6825
query15	250	169	167	167
query16	7944	269	264	264
query17	1817	523	517	517
query18	2021	284	270	270
query19	198	175	151	151
query20	93	88	85	85
query21	196	133	132	132
query22	4282	3980	4126	3980
query23	33714	33068	33061	33061
query24	11916	2793	2834	2793
query25	681	376	357	357
query26	1808	157	155	155
query27	2904	323	318	318
query28	7273	2094	2096	2094
query29	1093	623	599	599
query30	289	151	148	148
query31	956	783	779	779
query32	100	52	55	52
query33	760	261	266	261
query34	1002	483	470	470
query35	764	623	616	616
query36	1083	956	915	915
query37	174	66	65	65
query38	2866	2772	2797	2772
query39	850	798	804	798
query40	274	133	125	125
query41	46	45	43	43
query42	106	94	97	94
query43	597	588	571	571
query44	1275	737	746	737
query45	186	163	162	162
query46	1075	715	703	703
query47	1858	1763	1790	1763
query48	371	309	302	302
query49	1176	374	379	374
query50	766	405	398	398
query51	6986	6930	6628	6628
query52	104	91	90	90
query53	358	285	299	285
query54	1052	445	432	432
query55	75	72	73	72
query56	274	282	261	261
query57	1152	1026	1039	1026
query58	234	209	210	209
query59	3514	3311	3206	3206
query60	272	263	253	253
query61	92	93	91	91
query62	675	457	472	457
query63	309	285	281	281
query64	9752	2259	1746	1746
query65	3191	3103	3131	3103
query66	1394	342	325	325
query67	15051	15161	14588	14588
query68	4614	560	547	547
query69	437	271	263	263
query70	1187	1114	1137	1114
query71	389	272	271	271
query72	7613	5363	2713	2713
query73	714	323	327	323
query74	6103	5610	5550	5550
query75	3392	2646	2641	2641
query76	2629	987	1000	987
query77	414	266	264	264
query78	10337	9833	9738	9738
query79	1965	522	526	522
query80	1332	435	446	435
query81	516	223	219	219
query82	643	92	91	91
query83	283	168	169	168
query84	234	86	86	86
query85	1772	277	266	266
query86	491	305	305	305
query87	3325	3139	3081	3081
query88	4214	2438	2440	2438
query89	486	397	384	384
query90	1960	199	188	188
query91	132	106	106	106
query92	62	52	50	50
query93	2328	523	510	510
query94	1244	201	198	198
query95	412	317	323	317
query96	598	273	268	268
query97	3181	3069	3011	3011
query98	241	232	220	220
query99	1230	856	844	844
Total cold run time: 276920 ms
Total hot run time: 169541 ms
@doris-robot
Copy link

ClickBench: Total hot run time: 29.81 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 143776a229fd384a79ac4b99d45b04f6dfa2ecf7, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.22	0.06	0.05
query4	1.67	0.07	0.07
query5	0.51	0.47	0.50
query6	1.13	0.75	0.72
query7	0.01	0.01	0.02
query8	0.05	0.04	0.04
query9	0.54	0.50	0.50
query10	0.55	0.56	0.54
query11	0.16	0.11	0.11
query12	0.15	0.12	0.12
query13	0.58	0.58	0.59
query14	0.77	0.78	0.81
query15	0.83	0.81	0.80
query16	0.36	0.37	0.38
query17	1.03	1.02	1.04
query18	0.22	0.22	0.27
query19	1.90	1.73	1.69
query20	0.01	0.01	0.01
query21	15.52	0.70	0.69
query22	4.77	7.82	1.27
query23	18.28	1.38	1.24
query24	1.32	0.46	0.20
query25	0.16	0.08	0.08
query26	0.26	0.17	0.16
query27	0.08	0.08	0.08
query28	13.31	1.01	0.99
query29	12.69	3.26	3.31
query30	0.24	0.06	0.06
query31	2.88	0.37	0.38
query32	3.28	0.47	0.47
query33	2.91	2.93	2.85
query34	17.07	4.39	4.42
query35	4.57	4.48	4.48
query36	0.65	0.46	0.46
query37	0.18	0.16	0.15
query38	0.16	0.14	0.15
query39	0.05	0.03	0.04
query40	0.16	0.14	0.13
query41	0.10	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.03
Total cold run time: 109.55 s
Total hot run time: 29.81 s
@zddr
Copy link
Contributor Author

zddr commented May 28, 2024

run feut
1 similar comment
@zddr
Copy link
Contributor Author

zddr commented May 29, 2024

run feut
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label May 29, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.
@morrySnow morrySnow merged commit c1676f0 into apache:master May 29, 2024
25 of 26 checks passed
zddr added a commit to zddr/incubator-doris that referenced this pull request May 31, 2024
@zddr
[enhance](mtmv) Mv refresh on commit (apache#34548)
0c219ec 
support refresh MTMV when base table data change / drop
partition/replace partition

CREATE MATERIALIZED VIEW mv1
        REFRESH ON COMMIT
        AS
        SELECT xxx;
@zddr zddr mentioned this pull request May 31, 2024
Merged
morrySnow pushed a commit that referenced this pull request May 31, 2024
@zddr
[enhance](mtmv)Mv refresh on commit (#35702)
4414edd 
pick from master #34548

The modification involving CloudGlobalTransactionMgr was not picked up
to 2.1 because the 2.1 branch does not yet have the Thunderbolt
CloudGlobalTransactionMgr
dataroaring pushed a commit that referenced this pull request May 31, 2024
@zddr @dataroaring
[enhance](mtmv) Mv refresh on commit (#34548)
dd2ed39 
support refresh MTMV when base table data change / drop
partition/replace partition

CREATE MATERIALIZED VIEW mv1
        REFRESH ON COMMIT
        AS 
        SELECT xxx;
@gavinchou gavinchou mentioned this pull request Jun 13, 2024
Merged
@morningman morningman mentioned this pull request Jun 22, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.4-merged kind/need-document reviewed
4 participants
@zddr @doris-robot @mymeiyi @morrySnow