Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](index) should not use light index change for bloom filter index #34732

Open
wants to merge 1 commit into
base: branch-2.0
Choose a base branch
from

Conversation

cambyzju
Copy link
Contributor

@cambyzju cambyzju commented May 11, 2024

cherry-pick #35214 to branch2.0

Problem:

  1. add or drop Bitmap Index;
  2. add or drop Bloom Filter Index;
    Both do not work correctly for history data.

related pr: #28135 #32839 #10136

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add regression cases.

@cambyzju cambyzju changed the title [fix](index) add index and drop index do not work for history data [fix](index) should not use light index change for bloom filter index May 22, 2024
@cambyzju
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 49862 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 764f13438b4ca87edfdf593a47767c7d1d3275c9, data reload: false

------ Round 1 ----------------------------------
q1	17720	4415	4361	4361
q2	2065	157	151	151
q3	10538	1913	1957	1913
q4	10336	1258	1326	1258
q5	8608	3979	3913	3913
q6	234	121	124	121
q7	2041	1571	1574	1571
q8	9583	2750	2712	2712
q9	14081	10405	10456	10405
q10	8638	3554	3532	3532
q11	418	250	253	250
q12	468	299	293	293
q13	18369	3944	4043	3944
q14	349	325	334	325
q15	509	457	461	457
q16	668	563	569	563
q17	1157	984	951	951
q18	7341	6791	6870	6791
q19	1733	1559	1594	1559
q20	548	304	315	304
q21	4456	4133	4111	4111
q22	490	380	377	377
Total cold run time: 120350 ms
Total hot run time: 49862 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4298	4253	4268	4253
q2	313	217	222	217
q3	4137	4181	4139	4139
q4	2758	2743	2739	2739
q5	7216	7135	7126	7126
q6	235	118	119	118
q7	3221	2807	2788	2788
q8	4348	4441	4459	4441
q9	17369	17003	16882	16882
q10	4236	4253	4270	4253
q11	745	644	682	644
q12	1017	849	851	849
q13	6787	3763	3741	3741
q14	456	429	427	427
q15	494	463	448	448
q16	738	679	689	679
q17	3720	3863	3825	3825
q18	8929	8832	8758	8758
q19	1742	1692	1679	1679
q20	2417	2112	2099	2099
q21	8457	8281	8443	8281
q22	1004	970	959	959
Total cold run time: 84637 ms
Total hot run time: 79345 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.82% (8082/21372)
Line Coverage: 29.47% (65992/223941)
Region Coverage: 28.94% (33982/117425)
Branch Coverage: 24.80% (17442/70338)
Coverage Report: http://coverage.selectdb-in.cc/coverage/764f13438b4ca87edfdf593a47767c7d1d3275c9_764f13438b4ca87edfdf593a47767c7d1d3275c9/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 203796 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 764f13438b4ca87edfdf593a47767c7d1d3275c9, data reload: false

query1	932	382	375	375
query2	6690	2704	2626	2626
query3	6923	212	210	210
query4	20429	17890	18015	17890
query5	19719	6507	6519	6507
query6	289	222	238	222
query7	4161	297	299	297
query8	272	229	247	229
query9	3126	2700	2599	2599
query10	425	293	294	293
query11	11422	10712	10777	10712
query12	119	78	72	72
query13	5597	689	694	689
query14	17813	13611	13362	13362
query15	360	215	221	215
query16	6455	273	259	259
query17	1754	1467	891	891
query18	2292	400	406	400
query19	211	150	153	150
query20	84	79	80	79
query21	192	105	93	93
query22	5233	5094	4973	4973
query23	32452	31908	31793	31793
query24	6913	6560	6515	6515
query25	507	427	431	427
query26	530	160	156	156
query27	1892	295	291	291
query28	6168	2342	2310	2310
query29	2883	2755	2856	2755
query30	243	165	162	162
query31	905	727	722	722
query32	65	60	64	60
query33	403	246	249	246
query34	875	470	483	470
query35	1123	881	944	881
query36	1334	1097	1182	1097
query37	93	58	64	58
query38	3113	2923	2906	2906
query39	1385	1310	1325	1310
query40	207	95	93	93
query41	38	40	36	36
query42	78	86	77	77
query43	742	760	649	649
query44	1164	710	718	710
query45	240	230	230	230
query46	1228	951	997	951
query47	1864	1702	1670	1670
query48	1007	707	709	707
query49	612	366	376	366
query50	858	608	598	598
query51	4772	4696	4637	4637
query52	88	83	75	75
query53	445	317	313	313
query54	2663	2464	2463	2463
query55	85	88	77	77
query56	212	223	217	217
query57	1149	1062	1099	1062
query58	222	203	199	199
query59	4011	4120	3942	3942
query60	205	194	211	194
query61	90	87	85	85
query62	886	469	507	469
query63	477	342	343	342
query64	2531	1521	1506	1506
query65	3593	3551	3563	3551
query66	807	388	365	365
query67	15866	15891	17196	15891
query68	8861	670	658	658
query69	583	353	364	353
query70	1588	1344	1449	1344
query71	413	306	305	305
query72	6459	3459	3474	3459
query73	741	321	328	321
query74	6355	5818	5849	5818
query75	5275	3720	3690	3690
query76	5297	1160	1202	1160
query77	898	256	250	250
query78	12726	13144	11972	11972
query79	9244	635	648	635
query80	808	391	378	378
query81	484	233	231	231
query82	370	95	100	95
query83	167	130	134	130
query84	259	70	69	69
query85	692	299	289	289
query86	342	308	315	308
query87	3261	2953	3053	2953
query88	4731	2337	2350	2337
query89	425	276	279	276
query90	1883	210	209	209
query91	167	134	134	134
query92	59	50	51	50
query93	4162	581	562	562
query94	698	202	206	202
query95	1096	1049	1053	1049
query96	648	333	322	322
query97	6608	6387	6433	6387
query98	190	174	177	174
query99	2757	915	899	899
Total cold run time: 309769 ms
Total hot run time: 203796 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.61 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 764f13438b4ca87edfdf593a47767c7d1d3275c9, data reload: false

query1	0.02	0.02	0.02
query2	0.08	0.02	0.02
query3	0.26	0.05	0.05
query4	1.77	0.06	0.06
query5	0.53	0.53	0.52
query6	1.25	0.62	0.62
query7	0.01	0.01	0.01
query8	0.03	0.03	0.02
query9	0.52	0.48	0.48
query10	0.54	0.54	0.53
query11	0.12	0.08	0.09
query12	0.12	0.09	0.09
query13	0.62	0.62	0.60
query14	0.79	0.77	0.79
query15	0.79	0.76	0.76
query16	0.36	0.37	0.38
query17	1.02	0.96	1.02
query18	0.21	0.27	0.24
query19	1.87	1.82	1.86
query20	0.01	0.01	0.01
query21	15.47	0.56	0.57
query22	2.14	2.04	1.54
query23	16.89	0.87	0.92
query24	7.15	1.32	0.98
query25	0.40	0.11	0.04
query26	0.71	0.15	0.16
query27	0.05	0.03	0.03
query28	5.70	0.78	0.72
query29	12.63	2.42	2.34
query30	0.54	0.49	0.53
query31	2.83	0.38	0.38
query32	3.37	0.50	0.50
query33	3.05	3.05	3.09
query34	15.24	4.81	4.81
query35	4.87	4.85	4.84
query36	1.06	1.02	1.01
query37	0.06	0.05	0.05
query38	0.04	0.02	0.03
query39	0.02	0.01	0.02
query40	0.16	0.15	0.14
query41	0.07	0.01	0.01
query42	0.02	0.02	0.02
query43	0.02	0.02	0.02
Total cold run time: 103.41 s
Total hot run time: 30.61 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 764f13438b4ca87edfdf593a47767c7d1d3275c9 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       21.6 seconds inserted 10000000 Rows, about 462K ops/s

Copy link
Contributor

@lide-reed lide-reed left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants