Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](load) fix wrong assert and cancel load error #35352

Merged
merged 2 commits into from
May 27, 2024

Conversation

morningman
Copy link
Contributor

@morningman morningman commented May 24, 2024

Proposed changes

java.lang.IllegalArgumentException: null
        at com.google.common.base.Preconditions.checkArgument(Preconditions.java:129) ~[guava-32.1.2-jre.jar:?]
        at org.apache.doris.qe.Coordinator.updateFragmentExecStatus(Coordinator.java:2590) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.QeProcessorImpl.reportExecStatus(QeProcessorImpl.java:253) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.service.FrontendServiceImpl.reportExecStatus(FrontendServiceImpl.java:960) ~[doris-fe.jar:1.2-SNAPSHOT]
        at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source) ~[?:?]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_352-352]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_352-352]
        at org.apache.doris.service.FeServer.lambda$start$0(FeServer.java:60) ~[doris-fe.jar:1.2-SNAPSHOT]
        at com.sun.proxy.$Proxy26.reportExecStatus(Unknown Source) ~[?:?]
        at org.apache.doris.thrift.FrontendService$Processor$reportExecStatus.getResult(FrontendService.java:3632) ~[fe-common-1.2-SNAPSHOT.jar:1.2-SNAPSHOT]
        at org.apache.doris.thrift.FrontendService$Processor$reportExecStatus.getResult(FrontendService.java:3612) ~[fe-common-1.2-SNAPSHOT.jar:1.2-SNAPSHOT]
        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38) ~[libthrift-0.16.0.jar:0.16.0]
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38) ~[libthrift-0.16.0.jar:0.16.0]
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:250) ~[libthrift-0.16.0.jar:0.16.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_352-352]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_352-352]
        at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_352-352]
  1. Fix unable to cancel load

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@morningman
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40635 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2dc368abc04120b632233b939b0d4de60022bd7f, data reload: false

------ Round 1 ----------------------------------
q1	17801	4614	4309	4309
q2	2028	183	195	183
q3	10562	1233	1147	1147
q4	10160	837	801	801
q5	7464	2711	2640	2640
q6	215	131	132	131
q7	954	609	599	599
q8	9250	2098	2101	2098
q9	9046	6555	6539	6539
q10	8817	3711	3717	3711
q11	468	238	241	238
q12	425	218	223	218
q13	17778	3001	2995	2995
q14	261	223	212	212
q15	508	474	468	468
q16	502	377	385	377
q17	974	659	743	659
q18	8220	7597	7366	7366
q19	5895	1562	1491	1491
q20	648	297	303	297
q21	4985	3883	3958	3883
q22	341	273	284	273
Total cold run time: 117302 ms
Total hot run time: 40635 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4435	4233	4286	4233
q2	381	277	271	271
q3	3018	2794	2772	2772
q4	1900	1598	1664	1598
q5	5307	5310	5274	5274
q6	216	126	129	126
q7	2153	1814	1758	1758
q8	3218	3323	3337	3323
q9	8426	8393	8346	8346
q10	3902	3647	3707	3647
q11	591	480	486	480
q12	758	595	594	594
q13	16397	3005	3000	3000
q14	286	245	259	245
q15	537	482	471	471
q16	464	425	432	425
q17	1800	1495	1472	1472
q18	7681	7663	7566	7566
q19	1661	1520	1554	1520
q20	1978	1783	1813	1783
q21	4966	4741	4775	4741
q22	571	477	499	477
Total cold run time: 70646 ms
Total hot run time: 54122 ms
@doris-robot
Copy link

TPC-DS: Total hot run time: 172134 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2dc368abc04120b632233b939b0d4de60022bd7f, data reload: false

query1	915	375	368	368
query2	6463	2397	2237	2237
query3	6658	205	206	205
query4	19240	17443	17245	17245
query5	4156	413	406	406
query6	248	152	152	152
query7	4586	299	286	286
query8	234	182	180	180
query9	8558	2390	2383	2383
query10	454	304	264	264
query11	10606	10098	10070	10070
query12	141	89	91	89
query13	1651	365	366	365
query14	9492	7597	7255	7255
query15	235	170	173	170
query16	7898	278	260	260
query17	1838	540	525	525
query18	1991	285	275	275
query19	201	160	166	160
query20	94	92	88	88
query21	195	138	126	126
query22	4150	3873	3915	3873
query23	33535	33063	32892	32892
query24	6659	2763	2742	2742
query25	532	364	351	351
query26	710	153	155	153
query27	1907	316	314	314
query28	3794	2072	2067	2067
query29	850	611	592	592
query30	247	173	176	173
query31	981	765	736	736
query32	94	51	51	51
query33	521	261	258	258
query34	845	469	471	469
query35	695	589	576	576
query36	1046	902	877	877
query37	110	69	70	69
query38	2905	2760	2762	2760
query39	860	788	785	785
query40	192	123	121	121
query41	46	45	43	43
query42	101	97	91	91
query43	587	554	541	541
query44	1083	732	750	732
query45	183	165	160	160
query46	1057	716	722	716
query47	1847	1746	1759	1746
query48	393	293	290	290
query49	771	396	374	374
query50	778	375	378	375
query51	6837	6828	6685	6685
query52	96	91	89	89
query53	343	283	279	279
query54	528	436	413	413
query55	72	72	70	70
query56	262	235	267	235
query57	1126	1044	1023	1023
query58	236	204	209	204
query59	3235	3055	3268	3055
query60	278	258	301	258
query61	89	92	86	86
query62	601	464	441	441
query63	308	288	279	279
query64	8462	2184	1715	1715
query65	3158	3100	3114	3100
query66	802	334	321	321
query67	15250	15131	14934	14934
query68	4780	542	538	538
query69	483	270	266	266
query70	1156	1098	1137	1098
query71	419	262	289	262
query72	8070	5572	5633	5572
query73	730	321	318	318
query74	6089	5564	5659	5564
query75	3451	2642	2632	2632
query76	3020	955	1069	955
query77	670	261	261	261
query78	10339	9885	10063	9885
query79	2105	516	514	514
query80	756	440	422	422
query81	500	244	248	244
query82	1090	89	90	89
query83	227	211	163	163
query84	234	80	84	80
query85	1110	267	273	267
query86	489	307	307	307
query87	3291	3133	3145	3133
query88	3997	2351	2329	2329
query89	476	392	382	382
query90	2038	184	181	181
query91	122	95	94	94
query92	61	46	47	46
query93	1971	522	501	501
query94	1148	180	181	180
query95	400	305	307	305
query96	591	270	264	264
query97	3133	2997	3016	2997
query98	229	226	213	213
query99	1188	859	849	849
Total cold run time: 259379 ms
Total hot run time: 172134 ms
@doris-robot
Copy link

ClickBench: Total hot run time: 31.1 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2dc368abc04120b632233b939b0d4de60022bd7f, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.04
query4	1.68	0.07	0.08
query5	0.49	0.50	0.50
query6	1.13	0.73	0.72
query7	0.02	0.02	0.01
query8	0.05	0.03	0.04
query9	0.52	0.48	0.48
query10	0.55	0.56	0.55
query11	0.16	0.10	0.11
query12	0.15	0.12	0.12
query13	0.60	0.59	0.60
query14	0.77	0.78	0.80
query15	0.83	0.81	0.82
query16	0.35	0.36	0.36
query17	0.98	1.02	0.94
query18	0.21	0.21	0.26
query19	1.75	1.70	1.64
query20	0.02	0.01	0.01
query21	15.44	0.68	0.66
query22	4.83	5.82	2.59
query23	18.26	1.31	1.27
query24	1.66	0.32	0.20
query25	0.16	0.08	0.08
query26	0.26	0.16	0.17
query27	0.07	0.08	0.07
query28	13.32	1.03	1.01
query29	13.23	3.31	3.28
query30	0.24	0.05	0.06
query31	2.91	0.39	0.38
query32	3.27	0.47	0.47
query33	2.90	2.88	2.88
query34	17.10	4.45	4.41
query35	4.53	4.52	4.65
query36	0.65	0.46	0.46
query37	0.17	0.16	0.15
query38	0.16	0.15	0.14
query39	0.04	0.04	0.03
query40	0.17	0.14	0.14
query41	0.09	0.04	0.04
query42	0.05	0.04	0.04
query43	0.04	0.04	0.04
Total cold run time: 110.16 s
Total hot run time: 31.1 s
@morningman morningman marked this pull request as ready for review May 26, 2024 05:59
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels May 26, 2024
Copy link
Contributor

PR approved by anyone and no changes requested.

Copy link
Contributor

@kaka11chen kaka11chen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morningman morningman merged commit 592a6e0 into apache:master May 27, 2024
28 of 30 checks passed
dataroaring pushed a commit that referenced this pull request May 27, 2024
1.
```
java.lang.IllegalArgumentException: null
        at com.google.common.base.Preconditions.checkArgument(Preconditions.java:129) ~[guava-32.1.2-jre.jar:?]
        at org.apache.doris.qe.Coordinator.updateFragmentExecStatus(Coordinator.java:2590) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.QeProcessorImpl.reportExecStatus(QeProcessorImpl.java:253) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.service.FrontendServiceImpl.reportExecStatus(FrontendServiceImpl.java:960) ~[doris-fe.jar:1.2-SNAPSHOT]
        at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source) ~[?:?]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_352-352]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_352-352]
        at org.apache.doris.service.FeServer.lambda$start$0(FeServer.java:60) ~[doris-fe.jar:1.2-SNAPSHOT]
        at com.sun.proxy.$Proxy26.reportExecStatus(Unknown Source) ~[?:?]
        at org.apache.doris.thrift.FrontendService$Processor$reportExecStatus.getResult(FrontendService.java:3632) ~[fe-common-1.2-SNAPSHOT.jar:1.2-SNAPSHOT]
        at org.apache.doris.thrift.FrontendService$Processor$reportExecStatus.getResult(FrontendService.java:3612) ~[fe-common-1.2-SNAPSHOT.jar:1.2-SNAPSHOT]
        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38) ~[libthrift-0.16.0.jar:0.16.0]
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38) ~[libthrift-0.16.0.jar:0.16.0]
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:250) ~[libthrift-0.16.0.jar:0.16.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_352-352]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_352-352]
        at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_352-352]
```

2. Fix unable to cancel load
seawinde pushed a commit to seawinde/doris that referenced this pull request May 27, 2024
1.
```
java.lang.IllegalArgumentException: null
        at com.google.common.base.Preconditions.checkArgument(Preconditions.java:129) ~[guava-32.1.2-jre.jar:?]
        at org.apache.doris.qe.Coordinator.updateFragmentExecStatus(Coordinator.java:2590) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.QeProcessorImpl.reportExecStatus(QeProcessorImpl.java:253) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.service.FrontendServiceImpl.reportExecStatus(FrontendServiceImpl.java:960) ~[doris-fe.jar:1.2-SNAPSHOT]
        at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source) ~[?:?]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_352-352]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_352-352]
        at org.apache.doris.service.FeServer.lambda$start$0(FeServer.java:60) ~[doris-fe.jar:1.2-SNAPSHOT]
        at com.sun.proxy.$Proxy26.reportExecStatus(Unknown Source) ~[?:?]
        at org.apache.doris.thrift.FrontendService$Processor$reportExecStatus.getResult(FrontendService.java:3632) ~[fe-common-1.2-SNAPSHOT.jar:1.2-SNAPSHOT]
        at org.apache.doris.thrift.FrontendService$Processor$reportExecStatus.getResult(FrontendService.java:3612) ~[fe-common-1.2-SNAPSHOT.jar:1.2-SNAPSHOT]
        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38) ~[libthrift-0.16.0.jar:0.16.0]
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38) ~[libthrift-0.16.0.jar:0.16.0]
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:250) ~[libthrift-0.16.0.jar:0.16.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_352-352]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_352-352]
        at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_352-352]
```

2. Fix unable to cancel load
@morningman morningman mentioned this pull request Jun 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.4-merged dev/3.0.0-merged p0_b reviewed
6 participants