Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](Nereids) remove restrict for count(*) in window #35220

Merged
merged 1 commit into from
May 23, 2024
Merged

[fix](Nereids) remove restrict for count(*) in window #35220

merged 1 commit into from
May 23, 2024

Conversation

morrySnow
Copy link
Contributor

@morrySnow morrySnow commented May 22, 2024 •
edited
Loading

support count(*) used for window function

CREATE TABLE `t1` (
  `id` INT NULL,
  `dt` TEXT NULL
)
DISTRIBUTED BY HASH(`id`) BUCKETS 10
PROPERTIES (
"replication_allocation" = "tag.location.default: 1" );

select *, count(*) over() from t1;

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...
@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.
@morrySnow
Copy link
Contributor Author

run buildall
@doris-robot
Copy link

TPC-DS: Total hot run time: 171594 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit eda1c1d15d12e9b4259aea6825a3b8bd958e1b97, data reload: false

query1	895	374	376	374
query2	6453	2392	2319	2319
query3	6653	208	204	204
query4	19015	17248	17206	17206
query5	4153	417	422	417
query6	245	157	175	157
query7	4590	301	297	297
query8	240	192	187	187
query9	8547	2415	2401	2401
query10	467	294	282	282
query11	10711	10081	10040	10040
query12	140	91	87	87
query13	1643	369	367	367
query14	9336	7706	6904	6904
query15	211	164	166	164
query16	7749	263	250	250
query17	1859	517	523	517
query18	1886	275	278	275
query19	203	170	157	157
query20	97	87	87	87
query21	193	132	124	124
query22	4217	3862	4083	3862
query23	33585	33043	32912	32912
query24	10977	2868	2901	2868
query25	651	357	355	355
query26	1501	160	158	158
query27	2995	320	329	320
query28	7518	2057	2049	2049
query29	982	622	596	596
query30	309	168	180	168
query31	1000	791	726	726
query32	101	54	55	54
query33	767	270	262	262
query34	1014	480	469	469
query35	770	600	595	595
query36	1092	913	930	913
query37	139	72	69	69
query38	2880	2775	2697	2697
query39	831	793	833	793
query40	277	129	124	124
query41	49	43	44	43
query42	103	96	101	96
query43	551	525	550	525
query44	1182	723	754	723
query45	179	164	160	160
query46	1074	717	718	717
query47	1838	1739	1791	1739
query48	379	296	294	294
query49	1117	386	379	379
query50	778	414	402	402
query51	6918	6714	6807	6714
query52	98	96	88	88
query53	353	295	293	293
query54	943	428	418	418
query55	75	74	72	72
query56	254	270	238	238
query57	1107	1077	1036	1036
query58	248	209	207	207
query59	3308	3165	3079	3079
query60	276	259	249	249
query61	88	86	90	86
query62	649	453	453	453
query63	312	287	293	287
query64	9879	2243	1751	1751
query65	3172	3101	3102	3101
query66	1397	332	318	318
query67	15324	14936	14820	14820
query68	4601	544	548	544
query69	452	268	274	268
query70	1152	1056	1085	1056
query71	424	265	274	265
query72	7266	5656	5163	5163
query73	725	322	329	322
query74	6068	5601	5628	5601
query75	3491	2620	2632	2620
query76	3046	1010	1036	1010
query77	618	269	276	269
query78	10551	9868	9708	9708
query79	2134	532	525	525
query80	931	471	498	471
query81	528	245	242	242
query82	639	97	92	92
query83	267	169	168	168
query84	241	86	85	85
query85	1724	265	281	265
query86	511	327	313	313
query87	3295	3126	3134	3126
query88	4042	2307	2317	2307
query89	493	400	387	387
query90	1995	190	188	188
query91	123	94	99	94
query92	59	50	49	49
query93	2303	521	496	496
query94	1237	251	195	195
query95	413	315	318	315
query96	585	268	272	268
query97	3161	3025	3038	3025
query98	234	222	217	217
query99	1115	861	846	846
Total cold run time: 272967 ms
Total hot run time: 171594 ms
@doris-robot
Copy link

ClickBench: Total hot run time: 30.14 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit eda1c1d15d12e9b4259aea6825a3b8bd958e1b97, data reload: false

query1	0.04	0.03	0.03
query2	0.09	0.03	0.04
query3	0.23	0.05	0.05
query4	1.68	0.07	0.07
query5	0.47	0.49	0.51
query6	1.12	0.73	0.72
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.54	0.49	0.49
query10	0.55	0.54	0.53
query11	0.16	0.11	0.11
query12	0.15	0.12	0.12
query13	0.59	0.60	0.60
query14	0.79	0.77	0.79
query15	0.82	0.82	0.81
query16	0.37	0.37	0.38
query17	0.99	0.99	0.93
query18	0.22	0.24	0.24
query19	1.90	1.70	1.74
query20	0.01	0.01	0.01
query21	15.46	0.70	0.67
query22	4.70	7.58	1.53
query23	18.31	1.47	1.34
query24	1.39	0.41	0.22
query25	0.16	0.07	0.08
query26	0.28	0.16	0.17
query27	0.09	0.07	0.09
query28	13.34	1.02	0.99
query29	12.73	3.28	3.26
query30	0.25	0.06	0.06
query31	2.86	0.39	0.38
query32	3.29	0.47	0.47
query33	2.94	2.86	2.93
query34	17.04	4.43	4.47
query35	4.53	4.48	4.51
query36	0.63	0.46	0.45
query37	0.18	0.15	0.16
query38	0.15	0.15	0.14
query39	0.04	0.03	0.04
query40	0.17	0.13	0.15
query41	0.09	0.04	0.05
query42	0.05	0.06	0.05
query43	0.03	0.04	0.04
Total cold run time: 109.5 s
Total hot run time: 30.14 s
@morrySnow
[fix](Nereids) remove restrict for count(*) in window
5abc96b 
support count(*) used for window function

CREATE TABLE `t1` (
  `id` INT NULL,
  `dt` TEXT NULL
)
DISTRIBUTED BY HASH(`id`) BUCKETS 10
PROPERTIES (
"replication_allocation" = "tag.location.default: 1"
);

select *, count(*) over() from t1;
@morrySnow
Copy link
Contributor Author

run buildall
@doris-robot
Copy link

TPC-DS: Total hot run time: 170071 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5abc96bfb3142a72849aae3d1c4c7674da1763e2, data reload: false

query1	923	376	364	364
query2	6457	2536	2382	2382
query3	6645	206	207	206
query4	19506	17445	17376	17376
query5	4177	422	436	422
query6	247	157	179	157
query7	4584	297	290	290
query8	241	184	195	184
query9	8491	2379	2365	2365
query10	461	286	272	272
query11	10510	10071	10142	10071
query12	147	92	91	91
query13	1653	367	367	367
query14	10167	7414	7556	7414
query15	232	174	171	171
query16	7775	272	266	266
query17	1837	541	520	520
query18	1814	287	284	284
query19	200	161	165	161
query20	100	87	88	87
query21	195	130	133	130
query22	4341	3908	3853	3853
query23	33734	33143	33119	33119
query24	11156	2935	2854	2854
query25	659	375	443	375
query26	1416	153	151	151
query27	2930	318	308	308
query28	7223	2033	2018	2018
query29	956	600	600	600
query30	306	173	171	171
query31	955	750	747	747
query32	88	55	54	54
query33	785	271	268	268
query34	989	480	477	477
query35	716	611	600	600
query36	1064	887	873	873
query37	130	68	73	68
query38	2901	2795	2743	2743
query39	824	769	801	769
query40	234	128	124	124
query41	46	44	44	44
query42	104	95	94	94
query43	553	541	549	541
query44	1234	723	746	723
query45	183	163	164	163
query46	1073	702	739	702
query47	1854	1770	1763	1763
query48	375	302	292	292
query49	1083	367	374	367
query50	763	405	388	388
query51	6864	6789	6744	6744
query52	105	87	95	87
query53	348	286	282	282
query54	965	431	416	416
query55	74	71	72	71
query56	270	248	241	241
query57	1142	1033	1022	1022
query58	244	211	210	210
query59	3364	3125	3252	3125
query60	273	255	251	251
query61	87	86	87	86
query62	685	451	447	447
query63	302	284	281	281
query64	9946	2227	1744	1744
query65	3174	3104	3126	3104
query66	1384	333	332	332
query67	15435	14933	14890	14890
query68	6203	533	538	533
query69	495	268	265	265
query70	1153	1140	1111	1111
query71	485	275	278	275
query72	7989	2725	2528	2528
query73	799	321	322	321
query74	6157	5697	5659	5659
query75	4029	2643	2632	2632
query76	4325	985	992	985
query77	662	268	267	267
query78	10413	10032	9751	9751
query79	2756	509	508	508
query80	1343	443	444	443
query81	526	250	243	243
query82	1423	105	97	97
query83	208	172	170	170
query84	265	86	87	86
query85	1440	269	262	262
query86	476	330	296	296
query87	3310	3118	3114	3114
query88	3955	2330	2310	2310
query89	480	383	384	383
query90	1969	191	188	188
query91	124	98	98	98
query92	64	48	48	48
query93	2972	521	500	500
query94	1149	192	186	186
query95	389	305	311	305
query96	597	266	270	266
query97	3206	3053	3007	3007
query98	230	224	225	224
query99	1230	886	848	848
Total cold run time: 280521 ms
Total hot run time: 170071 ms
@doris-robot
Copy link

ClickBench: Total hot run time: 31.3 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5abc96bfb3142a72849aae3d1c4c7674da1763e2, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.23	0.06	0.06
query4	1.67	0.11	0.08
query5	0.49	0.50	0.51
query6	1.12	0.73	0.73
query7	0.02	0.02	0.01
query8	0.04	0.04	0.04
query9	0.53	0.49	0.48
query10	0.54	0.55	0.54
query11	0.16	0.11	0.11
query12	0.15	0.12	0.13
query13	0.61	0.59	0.60
query14	0.75	0.79	0.78
query15	0.84	0.82	0.82
query16	0.36	0.35	0.37
query17	0.98	1.01	1.00
query18	0.22	0.25	0.27
query19	1.86	1.71	1.76
query20	0.02	0.01	0.01
query21	15.52	0.70	0.69
query22	4.00	7.09	2.55
query23	18.26	1.39	1.23
query24	1.80	0.20	0.26
query25	0.15	0.09	0.08
query26	0.25	0.16	0.16
query27	0.08	0.07	0.08
query28	13.32	1.04	1.00
query29	12.68	3.28	3.29
query30	0.25	0.06	0.06
query31	2.86	0.40	0.39
query32	3.26	0.48	0.47
query33	2.91	2.90	2.91
query34	16.95	4.43	4.42
query35	4.52	4.50	4.51
query36	0.66	0.46	0.46
query37	0.17	0.15	0.16
query38	0.15	0.15	0.14
query39	0.04	0.04	0.04
query40	0.17	0.15	0.13
query41	0.09	0.05	0.04
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 108.9 s
Total hot run time: 31.3 s
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.
@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels May 23, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.
@morrySnow morrySnow merged commit d8c39a9 into apache:master May 23, 2024
26 of 28 checks passed
@morrySnow morrySnow deleted the support_count_star_window branch May 23, 2024 02:46
yiguolei pushed a commit that referenced this pull request May 24, 2024
@morrySnow @Doris-Extras
[opt](Nereids) remove restrict for count(*) in window (#35220)
98b2bda 
support count(*) used for window function

CREATE TABLE `t1` (
  `id` INT NULL,
  `dt` TEXT NULL
)
DISTRIBUTED BY HASH(`id`) BUCKETS 10
PROPERTIES (
"replication_allocation" = "tag.location.default: 1"
);

select *, count(*) over() from t1;
dataroaring pushed a commit that referenced this pull request May 26, 2024
@morrySnow @dataroaring
[opt](Nereids) remove restrict for count(*) in window (#35220)
362ee9f 
support count(*) used for window function

CREATE TABLE `t1` (
  `id` INT NULL,
  `dt` TEXT NULL
)
DISTRIBUTED BY HASH(`id`) BUCKETS 10
PROPERTIES (
"replication_allocation" = "tag.location.default: 1"
);

select *, count(*) over() from t1;
morrySnow added a commit to morrySnow/incubator-doris that referenced this pull request May 27, 2024
@morrySnow
[opt](Nereids) remove restrict for count(*) in window (apache#35220)
53b6617 
pick from master apache#35220

support count(*) used for window function

CREATE TABLE `t1` (
  `id` INT NULL,
  `dt` TEXT NULL
)
DISTRIBUTED BY HASH(`id`) BUCKETS 10
PROPERTIES (
"replication_allocation" = "tag.location.default: 1"
);

select *, count(*) over() from t1;
@morrySnow morrySnow mentioned this pull request May 27, 2024
Merged
morrySnow added a commit that referenced this pull request May 27, 2024
@morrySnow
[opt](Nereids) remove restrict for count(*) in window (#35220) (#35452)
f8ea072 
pick from master #35220

support count(*) used for window function

CREATE TABLE `t1` (
  `id` INT NULL,
  `dt` TEXT NULL
)
DISTRIBUTED BY HASH(`id`) BUCKETS 10
PROPERTIES (
"replication_allocation" = "tag.location.default: 1"
);

select *, count(*) over() from t1;
@morningman morningman mentioned this pull request Jun 22, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.11-merged dev/2.1.4-merged reviewed
4 participants
@morrySnow @doris-robot @924060929 @starocean999