Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature](topn) BE adaptive choose whether push topn filter down to storage layer #34713

Merged
merged 1 commit into from May 13, 2024

Conversation

BiteTheDDDDt
Copy link
Contributor

@BiteTheDDDDt BiteTheDDDDt commented May 11, 2024

Proposed changes

support judge topn filter push down
topn filter will push down to storage layer when 2 case:

  1. filter target is key column
  2. table data model is dup/merge on write

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@BiteTheDDDDt
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.65% (8981/25191)
Line Coverage: 27.32% (74253/271743)
Region Coverage: 26.56% (38384/144508)
Branch Coverage: 23.38% (19574/83722)
Coverage Report: http://coverage.selectdb-in.cc/coverage/07050d324203e91c944dff99703c342e8f4de93f_07050d324203e91c944dff99703c342e8f4de93f/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 40599 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 07050d324203e91c944dff99703c342e8f4de93f, data reload: false

------ Round 1 ----------------------------------
q1	17696	4615	4333	4333
q2	2601	190	197	190
q3	11072	1169	1192	1169
q4	10307	794	754	754
q5	7535	2707	2681	2681
q6	213	135	130	130
q7	1006	581	568	568
q8	9299	2135	2061	2061
q9	9022	6568	6557	6557
q10	8910	3701	3736	3701
q11	456	236	233	233
q12	500	218	215	215
q13	17764	2972	2938	2938
q14	255	215	222	215
q15	507	480	479	479
q16	515	385	381	381
q17	984	690	692	690
q18	8066	7617	7323	7323
q19	3305	1544	1454	1454
q20	659	325	316	316
q21	5005	3935	3975	3935
q22	361	288	276	276
Total cold run time: 116038 ms
Total hot run time: 40599 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4307	4218	4204	4204
q2	367	271	267	267
q3	2972	2768	2722	2722
q4	1895	1626	1582	1582
q5	5274	5286	5294	5286
q6	211	124	124	124
q7	2210	1884	1884	1884
q8	3211	3335	3332	3332
q9	8394	8394	8325	8325
q10	3847	3653	3646	3646
q11	580	485	488	485
q12	758	606	603	603
q13	17492	2938	2923	2923
q14	289	244	249	244
q15	517	494	484	484
q16	458	410	437	410
q17	1779	1485	1474	1474
q18	7644	7448	7454	7448
q19	1697	1550	1543	1543
q20	1940	1779	1752	1752
q21	4870	4854	4690	4690
q22	568	487	497	487
Total cold run time: 71280 ms
Total hot run time: 53915 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 186739 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 07050d324203e91c944dff99703c342e8f4de93f, data reload: false

query1	913	370	344	344
query2	6451	2358	2366	2358
query3	6657	215	218	215
query4	25589	21231	21123	21123
query5	4133	416	420	416
query6	271	170	173	170
query7	4590	287	281	281
query8	239	184	193	184
query9	8492	2411	2395	2395
query10	442	260	260	260
query11	14649	14234	14202	14202
query12	137	90	94	90
query13	1639	371	375	371
query14	9839	7571	8413	7571
query15	247	171	177	171
query16	8175	277	262	262
query17	1845	568	567	567
query18	2097	289	276	276
query19	265	151	156	151
query20	102	88	86	86
query21	193	128	128	128
query22	4997	4884	4837	4837
query23	34185	33572	33460	33460
query24	12072	2863	2900	2863
query25	675	382	377	377
query26	1789	159	153	153
query27	3020	335	320	320
query28	7691	2074	2042	2042
query29	1017	607	623	607
query30	295	145	150	145
query31	948	764	750	750
query32	91	54	52	52
query33	753	247	245	245
query34	1059	474	493	474
query35	809	673	687	673
query36	1088	881	932	881
query37	280	67	68	67
query38	2904	2787	2728	2728
query39	1683	1548	1561	1548
query40	269	125	120	120
query41	42	38	39	38
query42	103	94	94	94
query43	608	553	559	553
query44	1224	713	729	713
query45	264	256	254	254
query46	1085	709	712	709
query47	1943	1900	1945	1900
query48	365	302	285	285
query49	1178	386	404	386
query50	774	390	376	376
query51	6867	6815	6807	6807
query52	105	99	86	86
query53	352	276	280	276
query54	976	436	438	436
query55	78	71	70	70
query56	239	214	220	214
query57	1238	1139	1160	1139
query58	222	194	193	193
query59	3395	3209	3151	3151
query60	252	234	234	234
query61	88	126	87	87
query62	692	466	476	466
query63	304	280	281	280
query64	9724	7393	7396	7393
query65	3146	3073	3094	3073
query66	1388	343	331	331
query67	15544	15202	14970	14970
query68	4658	518	521	518
query69	472	303	298	298
query70	1195	1112	1149	1112
query71	419	263	267	263
query72	7068	2521	2353	2353
query73	708	325	324	324
query74	6522	6106	6119	6106
query75	3482	2612	2629	2612
query76	2813	919	1039	919
query77	402	262	303	262
query78	10555	10126	9974	9974
query79	2744	504	510	504
query80	1324	430	430	430
query81	489	220	222	220
query82	1015	95	94	94
query83	195	170	169	169
query84	264	89	88	88
query85	1456	282	268	268
query86	466	298	305	298
query87	3316	3058	3095	3058
query88	4382	2397	2408	2397
query89	469	376	380	376
query90	2055	187	182	182
query91	121	101	95	95
query92	58	97	48	48
query93	3509	493	484	484
query94	1168	176	182	176
query95	406	305	299	299
query96	603	267	264	264
query97	3178	2962	2987	2962
query98	231	213	216	213
query99	1275	896	900	896
Total cold run time: 291804 ms
Total hot run time: 186739 ms

Copy link
Contributor

PR approved by anyone and no changes requested.

@englefly englefly changed the title [Feature](topn) support judge topn filter push down [Feature](topn) support judge topn filter push down to storage layer May 13, 2024
@englefly englefly changed the title [Feature](topn) support judge topn filter push down to storage layer [Feature](topn) BE adaptive choose whether push topn filter down to storage layer May 13, 2024
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label May 13, 2024
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@BiteTheDDDDt BiteTheDDDDt merged commit ab554be into apache:master May 13, 2024
28 of 32 checks passed
nextdreamblue pushed a commit to nextdreamblue/doris that referenced this pull request May 13, 2024
…torage layer (apache#34713)

support judge topn filter push down
topn filter will push down to storage layer when 2 case:

    filter target is key column
    table data model is dup/merge on write
M1saka2003 pushed a commit to M1saka2003/doris that referenced this pull request May 14, 2024
…torage layer (apache#34713)

support judge topn filter push down
topn filter will push down to storage layer when 2 case:

    filter target is key column
    table data model is dup/merge on write
ByteYue pushed a commit to ByteYue/doris that referenced this pull request May 15, 2024
…torage layer (apache#34713)

support judge topn filter push down
topn filter will push down to storage layer when 2 case:

    filter target is key column
    table data model is dup/merge on write
M1saka2003 pushed a commit to M1saka2003/doris that referenced this pull request May 24, 2024
…torage layer (apache#34713)

support judge topn filter push down
topn filter will push down to storage layer when 2 case:

    filter target is key column
    table data model is dup/merge on write
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants