Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](multi-table-load) fix be core when partial table load failed #34712

Merged
merged 1 commit into from May 20, 2024

Conversation

HHoflittlefish777
Copy link
Contributor

bug

When using single stream multi table load, be will core dump if partial table load failed.

reason

After some tables load fail, it will cause exec_task to return, leading to ctx destruction. When the execution plans of other tables are called back, it will cause the core.

solution

Wait all table plans exec finish.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@HHoflittlefish777 HHoflittlefish777 changed the title [fix](multi table load) fix be core when partial table load failed [fix](multi-table-load) fix be core when partial table load failed May 11, 2024
@HHoflittlefish777
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.65% (8981/25190)
Line Coverage: 27.33% (74268/271748)
Region Coverage: 26.56% (38384/144514)
Branch Coverage: 23.38% (19572/83722)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c98003649fe882cdaa317c76b3e8eaa282b026c1_c98003649fe882cdaa317c76b3e8eaa282b026c1/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 40694 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c98003649fe882cdaa317c76b3e8eaa282b026c1, data reload: false

------ Round 1 ----------------------------------
q1	6571	4296	4277	4277
q2	794	193	195	193
q3	6526	1129	1161	1129
q4	1005	722	751	722
q5	2651	2698	2643	2643
q6	231	167	151	151
q7	1094	598	572	572
q8	1976	2104	2107	2104
q9	7014	6916	6699	6699
q10	4142	3922	3800	3800
q11	360	260	247	247
q12	403	223	219	219
q13	16528	3202	3116	3116
q14	263	223	228	223
q15	525	468	483	468
q16	487	405	373	373
q17	989	692	703	692
q18	8254	7876	7639	7639
q19	4790	1541	1533	1533
q20	521	303	301	301
q21	5024	3312	4046	3312
q22	352	307	281	281
Total cold run time: 70500 ms
Total hot run time: 40694 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4423	4385	4334	4334
q2	375	256	264	256
q3	3019	2750	2697	2697
q4	1863	1576	1618	1576
q5	5287	5245	5294	5245
q6	208	120	119	119
q7	1716	1359	1376	1359
q8	3160	3337	3322	3322
q9	8372	8382	8380	8380
q10	3851	3651	3679	3651
q11	587	479	492	479
q12	757	590	602	590
q13	9039	2960	2963	2960
q14	276	269	251	251
q15	510	462	475	462
q16	471	415	411	411
q17	1796	1466	1503	1466
q18	7548	7566	7444	7444
q19	1671	1594	1554	1554
q20	1934	1793	1750	1750
q21	4908	4897	4613	4613
q22	566	519	462	462
Total cold run time: 62337 ms
Total hot run time: 53381 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 186514 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c98003649fe882cdaa317c76b3e8eaa282b026c1, data reload: false

query1	923	365	345	345
query2	6447	2438	2369	2369
query3	6640	211	216	211
query4	23024	21382	21343	21343
query5	4123	437	431	431
query6	251	164	168	164
query7	4592	292	283	283
query8	244	194	183	183
query9	8396	2459	2416	2416
query10	425	260	270	260
query11	14764	14087	14150	14087
query12	144	90	90	90
query13	1630	372	360	360
query14	10477	7606	8279	7606
query15	231	164	178	164
query16	7992	267	262	262
query17	1765	562	541	541
query18	2020	267	264	264
query19	186	146	144	144
query20	90	85	85	85
query21	192	131	130	130
query22	5022	4879	4865	4865
query23	34258	33479	33444	33444
query24	8064	2918	2904	2904
query25	565	358	360	358
query26	689	151	151	151
query27	2203	314	316	314
query28	5729	2061	2053	2053
query29	846	584	582	582
query30	245	158	154	154
query31	987	770	739	739
query32	87	52	54	52
query33	597	243	237	237
query34	882	477	473	473
query35	752	684	687	684
query36	1068	906	862	862
query37	100	63	64	63
query38	2883	2799	2743	2743
query39	1625	1574	1540	1540
query40	190	124	123	123
query41	43	39	38	38
query42	100	97	97	97
query43	551	553	532	532
query44	1043	711	725	711
query45	268	256	227	227
query46	1079	720	695	695
query47	1951	1916	1879	1879
query48	382	305	295	295
query49	858	395	411	395
query50	776	382	393	382
query51	6926	6816	6748	6748
query52	105	99	93	93
query53	345	288	281	281
query54	681	471	442	442
query55	74	73	72	72
query56	237	215	215	215
query57	1232	1146	1160	1146
query58	218	200	199	199
query59	3526	3010	3042	3010
query60	247	224	234	224
query61	87	89	86	86
query62	607	488	470	470
query63	308	284	282	282
query64	8502	7390	7321	7321
query65	3098	3059	3060	3059
query66	782	350	351	350
query67	15368	15047	14881	14881
query68	4544	524	529	524
query69	486	292	295	292
query70	1097	1053	1127	1053
query71	408	264	258	258
query72	7763	2511	2368	2368
query73	707	331	331	331
query74	6463	6078	6118	6078
query75	3317	2623	2628	2623
query76	2414	950	926	926
query77	484	265	259	259
query78	10621	9992	10263	9992
query79	2418	509	510	509
query80	1127	433	420	420
query81	518	217	228	217
query82	941	94	92	92
query83	231	167	164	164
query84	250	83	87	83
query85	1194	326	259	259
query86	452	317	301	301
query87	3260	3135	3119	3119
query88	4001	2416	2418	2416
query89	452	386	380	380
query90	2023	186	188	186
query91	124	96	98	96
query92	59	46	46	46
query93	1913	504	493	493
query94	1283	186	188	186
query95	393	307	306	306
query96	588	266	271	266
query97	3155	3015	2956	2956
query98	239	219	216	216
query99	1217	909	890	890
Total cold run time: 274884 ms
Total hot run time: 186514 ms

@HHoflittlefish777
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.65% (8984/25202)
Line Coverage: 27.32% (74273/271900)
Region Coverage: 26.54% (38375/144599)
Branch Coverage: 23.35% (19566/83782)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b794fc2b2672734e62e6443b3fd27298e0e99532_b794fc2b2672734e62e6443b3fd27298e0e99532/report/index.html

liaoxin01
liaoxin01 previously approved these changes May 14, 2024
Copy link
Contributor

@liaoxin01 liaoxin01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label May 14, 2024
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Copy link
Contributor

PR approved by anyone and no changes requested.

@HHoflittlefish777
Copy link
Contributor Author

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label May 19, 2024
@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.70% (9007/25228)
Line Coverage: 27.36% (74499/272290)
Region Coverage: 26.60% (38529/144830)
Branch Coverage: 23.43% (19660/83904)
Coverage Report: http://coverage.selectdb-in.cc/coverage/bfcbfa0b901bb519510d8be5dbf0c0ba52cab2bf_bfcbfa0b901bb519510d8be5dbf0c0ba52cab2bf/report/index.html

@HHoflittlefish777
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.71% (9008/25228)
Line Coverage: 27.37% (74515/272291)
Region Coverage: 26.60% (38526/144830)
Branch Coverage: 23.43% (19662/83904)
Coverage Report: http://coverage.selectdb-in.cc/coverage/71a846ab3f0324e2a240007247f1ba068c8d4b68_71a846ab3f0324e2a240007247f1ba068c8d4b68/report/index.html

Copy link
Contributor

@XuJianxu XuJianxu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label May 20, 2024
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@liaoxin01 liaoxin01 merged commit 1000e5b into apache:master May 20, 2024
23 of 28 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
2.1.0-conflict approved Indicates a PR has been approved by one committer. dev/2.0.x p0_c reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants