Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](nereids)set mark join reference for bitmap-in-apply #35435

Merged
merged 1 commit into from
May 27, 2024
Merged

[fix](nereids)set mark join reference for bitmap-in-apply #35435

merged 1 commit into from
May 27, 2024

Conversation

englefly
Copy link
Contributor

@englefly englefly commented May 27, 2024 •
edited
Loading

Proposed changes

bitmap filter is implemented before mark-join. When we support mark-join, author forgot to update the bitmap-filter branch.
when convert a bitmap-apply-in to join, we should set markjoinReference to the join if there are markJoinRefereneces

Issue Number: close #24085
#24085

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...
@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.
@englefly
Copy link
Contributor Author

run buildall
@morrySnow
Copy link
Contributor

add description and related issue id
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.
@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels May 27, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.
@doris-robot
Copy link

TPC-H: Total hot run time: 41893 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a2eb906c488c19ae5018c1a11cd41c4b7f3fc5ac, data reload: false

------ Round 1 ----------------------------------
q1	17604	4328	4232	4232
q2	2020	189	191	189
q3	10470	1264	1234	1234
q4	10186	774	826	774
q5	7491	2766	2730	2730
q6	217	135	135	135
q7	971	600	628	600
q8	9237	2140	2115	2115
q9	9510	6671	6702	6671
q10	9126	3896	3904	3896
q11	467	267	231	231
q12	470	214	217	214
q13	17183	3293	3208	3208
q14	264	226	226	226
q15	526	495	477	477
q16	513	419	398	398
q17	995	707	758	707
q18	8419	7999	7755	7755
q19	7769	1566	1563	1563
q20	1076	316	311	311
q21	5203	4044	3961	3961
q22	386	282	266	266
Total cold run time: 120103 ms
Total hot run time: 41893 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4578	4402	4343	4343
q2	514	280	263	263
q3	3142	2925	2921	2921
q4	1908	1548	1604	1548
q5	5390	5513	5523	5513
q6	216	119	123	119
q7	2157	1880	1824	1824
q8	3258	3413	3367	3367
q9	8643	8707	8764	8707
q10	4038	3706	3812	3706
q11	587	493	504	493
q12	797	623	611	611
q13	15956	3157	3193	3157
q14	305	277	296	277
q15	522	483	483	483
q16	505	438	450	438
q17	1856	1518	1508	1508
q18	7769	7661	7370	7370
q19	1655	1632	1540	1540
q20	1960	1806	1780	1780
q21	4804	4849	4744	4744
q22	571	490	470	470
Total cold run time: 71131 ms
Total hot run time: 55182 ms
@doris-robot
Copy link

TPC-DS: Total hot run time: 171776 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a2eb906c488c19ae5018c1a11cd41c4b7f3fc5ac, data reload: false

query1	912	381	363	363
query2	6431	2446	2363	2363
query3	6641	201	204	201
query4	19720	17319	17264	17264
query5	4172	416	431	416
query6	244	153	161	153
query7	4586	304	288	288
query8	237	198	180	180
query9	8493	2350	2330	2330
query10	451	288	254	254
query11	10501	10138	10022	10022
query12	137	88	87	87
query13	1644	387	372	372
query14	10259	7483	6888	6888
query15	225	171	166	166
query16	7749	272	265	265
query17	1584	540	553	540
query18	1956	282	288	282
query19	212	159	166	159
query20	103	86	89	86
query21	194	137	128	128
query22	4185	3988	3985	3985
query23	33719	33110	32898	32898
query24	6777	2930	2865	2865
query25	470	361	352	352
query26	686	152	156	152
query27	1940	321	319	319
query28	3406	2058	2044	2044
query29	850	597	592	592
query30	214	149	148	148
query31	947	743	729	729
query32	63	51	52	51
query33	517	261	292	261
query34	849	477	475	475
query35	704	625	599	599
query36	1079	916	932	916
query37	101	68	67	67
query38	2859	2763	2767	2763
query39	840	813	798	798
query40	197	127	125	125
query41	47	44	43	43
query42	100	93	94	93
query43	577	548	543	543
query44	1084	735	773	735
query45	179	154	159	154
query46	1051	715	715	715
query47	1806	1726	1780	1726
query48	363	302	310	302
query49	756	413	390	390
query50	758	379	378	378
query51	6756	6700	6657	6657
query52	101	95	90	90
query53	349	283	279	279
query54	538	438	422	422
query55	73	70	71	70
query56	251	242	240	240
query57	1111	1045	1042	1042
query58	231	200	201	200
query59	3551	3268	3216	3216
query60	290	258	255	255
query61	90	86	114	86
query62	548	452	448	448
query63	309	280	290	280
query64	2533	1722	1750	1722
query65	3153	3071	3115	3071
query66	787	327	329	327
query67	14947	14942	14723	14723
query68	4489	518	527	518
query69	435	267	262	262
query70	1118	1128	1146	1128
query71	381	268	269	268
query72	7830	5331	5535	5331
query73	748	330	321	321
query74	6000	5578	5602	5578
query75	3272	2585	2630	2585
query76	2188	1081	1062	1062
query77	376	268	266	266
query78	10331	9707	9745	9707
query79	2388	506	514	506
query80	1044	442	424	424
query81	536	225	217	217
query82	636	94	90	90
query83	251	176	168	168
query84	253	93	88	88
query85	1647	262	264	262
query86	537	306	294	294
query87	3297	3113	3117	3113
query88	4079	2456	2434	2434
query89	489	407	387	387
query90	2057	186	181	181
query91	125	98	97	97
query92	65	48	55	48
query93	2842	513	486	486
query94	1214	182	182	182
query95	388	304	313	304
query96	589	269	268	268
query97	3199	3007	2986	2986
query98	238	230	216	216
query99	1137	851	846	846
Total cold run time: 253372 ms
Total hot run time: 171776 ms
@doris-robot
Copy link

ClickBench: Total hot run time: 30.3 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a2eb906c488c19ae5018c1a11cd41c4b7f3fc5ac, data reload: false

query1	0.04	0.03	0.04
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.68	0.08	0.07
query5	0.49	0.49	0.51
query6	1.12	0.72	0.73
query7	0.02	0.01	0.02
query8	0.05	0.05	0.04
query9	0.54	0.50	0.49
query10	0.54	0.54	0.54
query11	0.16	0.11	0.10
query12	0.14	0.12	0.12
query13	0.58	0.59	0.59
query14	0.79	0.76	0.78
query15	0.82	0.81	0.81
query16	0.38	0.36	0.36
query17	0.99	0.97	0.98
query18	0.23	0.23	0.25
query19	1.78	1.80	1.80
query20	0.01	0.01	0.01
query21	15.59	0.66	0.65
query22	4.68	7.46	1.70
query23	18.28	1.29	1.21
query24	1.68	0.23	0.24
query25	0.15	0.08	0.08
query26	0.27	0.17	0.16
query27	0.09	0.08	0.08
query28	13.47	1.01	1.01
query29	13.68	3.27	3.26
query30	0.24	0.06	0.05
query31	2.87	0.39	0.38
query32	3.30	0.47	0.47
query33	2.95	2.84	2.90
query34	17.04	4.43	4.40
query35	4.50	4.50	4.63
query36	0.64	0.46	0.46
query37	0.17	0.15	0.15
query38	0.16	0.15	0.15
query39	0.04	0.04	0.04
query40	0.17	0.14	0.15
query41	0.09	0.04	0.05
query42	0.05	0.05	0.04
query43	0.04	0.04	0.03
Total cold run time: 110.82 s
Total hot run time: 30.3 s
@englefly
Copy link
Contributor Author

run buildall
@doris-robot
Copy link

TPC-H: Total hot run time: 39821 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a2eb906c488c19ae5018c1a11cd41c4b7f3fc5ac, data reload: false

------ Round 1 ----------------------------------
q1	18025	4417	4294	4294
q2	2592	197	193	193
q3	11292	1206	1176	1176
q4	10503	823	715	715
q5	8035	2718	2667	2667
q6	241	136	136	136
q7	959	625	610	610
q8	9260	2055	2044	2044
q9	8867	6457	6452	6452
q10	8882	3668	3670	3668
q11	452	240	234	234
q12	454	216	213	213
q13	17815	2956	2968	2956
q14	255	224	227	224
q15	501	464	471	464
q16	522	377	377	377
q17	973	693	713	693
q18	7996	7468	7505	7468
q19	6177	1448	1510	1448
q20	635	296	304	296
q21	4825	3226	3841	3226
q22	344	275	267	267
Total cold run time: 119605 ms
Total hot run time: 39821 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4365	4164	4232	4164
q2	371	266	277	266
q3	2996	2710	2723	2710
q4	1842	1562	1591	1562
q5	5186	5239	5244	5239
q6	205	124	125	124
q7	2140	1722	1756	1722
q8	3181	3321	3310	3310
q9	8350	8337	8290	8290
q10	3928	3693	3704	3693
q11	598	502	483	483
q12	770	603	630	603
q13	17112	2972	2979	2972
q14	300	264	266	264
q15	516	477	478	477
q16	480	408	416	408
q17	1746	1447	1456	1447
q18	7696	7578	7552	7552
q19	2435	1551	1530	1530
q20	1977	1776	1784	1776
q21	4841	4798	4703	4703
q22	566	516	489	489
Total cold run time: 71601 ms
Total hot run time: 53784 ms
@doris-robot
Copy link

TPC-DS: Total hot run time: 170756 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a2eb906c488c19ae5018c1a11cd41c4b7f3fc5ac, data reload: false

query1	917	376	371	371
query2	6463	2330	2366	2330
query3	6658	209	206	206
query4	19223	17408	17272	17272
query5	4162	416	410	410
query6	252	147	157	147
query7	4588	298	298	298
query8	231	190	175	175
query9	8423	2372	2346	2346
query10	445	296	249	249
query11	10397	10009	10097	10009
query12	138	89	90	89
query13	1639	362	370	362
query14	10104	6699	7560	6699
query15	209	167	166	166
query16	7818	259	252	252
query17	1857	514	510	510
query18	1964	267	275	267
query19	200	153	165	153
query20	98	87	90	87
query21	198	137	127	127
query22	4092	3881	3946	3881
query23	33338	33012	33026	33012
query24	11930	2801	2802	2801
query25	668	367	359	359
query26	1798	153	159	153
query27	3017	329	337	329
query28	7592	2053	2035	2035
query29	1157	611	582	582
query30	288	148	150	148
query31	950	727	729	727
query32	89	52	52	52
query33	758	261	288	261
query34	1010	474	478	474
query35	736	612	581	581
query36	1069	945	892	892
query37	277	71	70	70
query38	2903	2811	2785	2785
query39	844	777	779	777
query40	269	123	120	120
query41	46	43	45	43
query42	101	94	94	94
query43	574	544	531	531
query44	1230	725	753	725
query45	171	154	161	154
query46	1057	703	708	703
query47	1820	1806	1798	1798
query48	388	308	308	308
query49	1188	376	378	376
query50	776	376	396	376
query51	6921	6682	6806	6682
query52	103	88	85	85
query53	351	277	284	277
query54	1050	441	417	417
query55	74	75	70	70
query56	265	275	242	242
query57	1135	1029	1060	1029
query58	230	205	206	205
query59	3424	3252	3017	3017
query60	268	244	249	244
query61	87	87	84	84
query62	649	444	449	444
query63	312	277	277	277
query64	9749	2232	1691	1691
query65	3259	3102	3139	3102
query66	1385	328	328	328
query67	15341	15116	14695	14695
query68	4590	532	523	523
query69	436	263	266	263
query70	1157	1049	1072	1049
query71	409	273	267	267
query72	7660	5656	5216	5216
query73	714	319	325	319
query74	5952	5573	5526	5526
query75	3405	2675	2634	2634
query76	2750	1059	980	980
query77	403	265	255	255
query78	10029	9833	9594	9594
query79	2436	514	508	508
query80	1672	433	423	423
query81	507	221	216	216
query82	1253	90	93	90
query83	292	166	169	166
query84	248	88	86	86
query85	1570	262	261	261
query86	501	296	299	296
query87	3271	3057	3132	3057
query88	4065	2472	2459	2459
query89	480	387	373	373
query90	2007	185	180	180
query91	124	105	105	105
query92	62	50	49	49
query93	2176	516	492	492
query94	1238	193	257	193
query95	394	301	307	301
query96	596	270	270	270
query97	3163	3029	2983	2983
query98	235	228	216	216
query99	1222	845	873	845
Total cold run time: 275740 ms
Total hot run time: 170756 ms
@doris-robot
Copy link

ClickBench: Total hot run time: 30.94 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a2eb906c488c19ae5018c1a11cd41c4b7f3fc5ac, data reload: false

query1	0.04	0.03	0.04
query2	0.09	0.04	0.04
query3	0.22	0.06	0.06
query4	1.66	0.09	0.07
query5	0.50	0.49	0.50
query6	1.13	0.72	0.72
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.55	0.49	0.48
query10	0.53	0.54	0.53
query11	0.15	0.12	0.11
query12	0.14	0.13	0.12
query13	0.58	0.59	0.60
query14	0.76	0.79	0.78
query15	0.83	0.82	0.80
query16	0.35	0.36	0.34
query17	1.02	0.97	0.96
query18	0.21	0.25	0.25
query19	1.78	1.77	1.68
query20	0.01	0.02	0.01
query21	15.51	0.66	0.64
query22	3.36	6.56	2.54
query23	18.26	1.36	1.22
query24	1.92	0.24	0.21
query25	0.14	0.08	0.07
query26	0.26	0.17	0.16
query27	0.09	0.08	0.07
query28	13.32	1.02	0.99
query29	12.99	3.27	3.25
query30	0.24	0.06	0.05
query31	2.90	0.38	0.38
query32	3.27	0.47	0.47
query33	2.93	2.86	2.86
query34	17.09	4.40	4.44
query35	4.55	4.48	4.52
query36	0.65	0.46	0.48
query37	0.18	0.15	0.15
query38	0.16	0.14	0.14
query39	0.05	0.03	0.03
query40	0.16	0.14	0.14
query41	0.09	0.05	0.04
query42	0.05	0.05	0.05
query43	0.04	0.03	0.03
Total cold run time: 108.83 s
Total hot run time: 30.94 s
@englefly
Copy link
Contributor Author

run p0
@morrySnow morrySnow merged commit 8915744 into apache:master May 27, 2024
28 of 31 checks passed
@morrySnow morrySnow added the usercase Important user case type label label May 27, 2024
yiguolei pushed a commit that referenced this pull request May 28, 2024
@englefly @Doris-Extras
[fix](nereids) set mark join reference for bitmap-in-apply (#35435)
dfcabf8 
bitmap filter is implemented before mark-join. When support mark-join, we forgot to update the bitmap-filter branch.
when convert a bitmap-apply-in to join, we should set markjoinReference to the join if there are markJoinRefereneces
This was referenced May 28, 2024
Merged
Closed
morrySnow pushed a commit that referenced this pull request May 28, 2024
@englefly
[fix](nereids) set mark join reference for bitmap-in-apply (#35519)
ba82ca0 
pick #35435

bitmap filter is implemented before mark-join. When we support mark-join, author forgot to update the bitmap-filter branch.
when convert a bitmap-apply-in to join, we should set markjoinReference to the join if there are markJoinRefereneces
dataroaring pushed a commit that referenced this pull request May 28, 2024
@englefly @dataroaring
[fix](nereids) set mark join reference for bitmap-in-apply (#35435)
9766d9f 
bitmap filter is implemented before mark-join. When support mark-join, we forgot to update the bitmap-filter branch.
when convert a bitmap-apply-in to join, we should set markjoinReference to the join if there are markJoinRefereneces
@morningman morningman mentioned this pull request Jun 22, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.11-merged dev/2.1.4-merged reviewed usercase Important user case type label
4 participants
@englefly @doris-robot @morrySnow @starocean999