Skip to content

[improvement](segment) Skip nullable row ID gaps in sparse reads - #67800

Open
foxtail463 wants to merge 1 commit into
apache:masterfrom
foxtail463:improvement/skip-nullable-rowid-gaps
Open

[improvement](segment) Skip nullable row ID gaps in sparse reads#67800
foxtail463 wants to merge 1 commit into
apache:masterfrom
foxtail463:improvement/skip-nullable-rowid-gaps

Conversation

@foxtail463

Copy link
Copy Markdown
Contributor

Problem Summary:

Sparse row ID reads on nullable columns traversed every null-map run between selected rows, causing unnecessary
decoding work for large gaps.

Solution:

Add an optimized RleDecoder::Skip implementation that processes literal values in 64-bit chunks and counts
skipped nulls. Use it to advance null-map and data-decoder positions efficiently while preserving correct behavior for
both normal and null-map-only reads.

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@foxtail463

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 16720 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5b32913a519585871a9bebf50d821551674e67ca, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17574	3101	3119	3101
q2	2070	248	216	216
q3	10276	883	505	505
q4	4675	253	207	207
q5	7670	563	386	386
q6	138	116	96	96
q7	530	509	377	377
q8	9231	960	869	869
q9	3420	2409	2367	2367
q10	6527	870	711	711
q11	394	198	178	178
q12	610	263	199	199
q13	18172	1531	1160	1160
q14	162	152	135	135
q15	q16	439	398	370	370
q17	1350	941	826	826
q18	3089	2213	2251	2213
q19	1270	805	749	749
q20	383	279	206	206
q21	5616	1619	1843	1619
q22	320	266	230	230
Total cold run time: 93916 ms
Total hot run time: 16720 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3477	3397	3381	3381
q2	508	395	366	366
q3	2222	2303	2232	2232
q4	1178	1156	878	878
q5	2194	2079	2105	2079
q6	165	121	87	87
q7	1095	897	853	853
q8	1591	1395	1396	1395
q9	3117	3078	3063	3063
q10	1859	1785	1656	1656
q11	354	265	252	252
q12	448	433	336	336
q13	1446	1514	1179	1179
q14	178	169	156	156
q15	q16	393	395	357	357
q17	3601	3263	3180	3180
q18	4757	4365	4728	4365
q19	851	797	956	797
q20	1002	983	824	824
q21	3841	3120	3343	3120
q22	399	341	319	319
Total cold run time: 34676 ms
Total hot run time: 30875 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 79838 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5b32913a519585871a9bebf50d821551674e67ca, data reload: false

query5	4252	399	328	328
query6	403	134	124	124
query7	4952	387	184	184
query8	292	126	114	114
query9	8703	2827	2820	2820
query10	388	228	178	178
query11	5373	1028	924	924
query12	119	71	73	71
query13	1191	353	218	218
query14	6111	2188	2076	2076
query14_1	1965	1956	1974	1956
query15	191	120	110	110
query16	932	353	359	353
query17	757	406	335	335
query18	2322	297	214	214
query19	159	125	95	95
query20	75	68	67	67
query21	198	97	84	84
query22	5517	5359	5281	5281
query23	6438	6156	5975	5975
query23_1	6146	5943	5963	5943
query24	7299	1074	723	723
query24_1	726	709	709	709
query25	396	256	245	245
query26	1224	195	105	105
query27	2850	355	201	201
query28	4812	1255	1255	1255
query29	901	416	316	316
query30	246	150	125	125
query31	821	390	327	327
query32	127	70	70	70
query33	447	216	184	184
query34	981	788	455	455
query35	401	395	340	340
query36	587	574	531	531
query37	121	80	68	68
query38	1030	846	813	813
query39	487	475	465	465
query39_1	470	441	471	441
query40	206	84	68	68
query41	52	50	52	50
query42	67	64	66	64
query43	237	243	207	207
query44	965	388	410	388
query45	108	108	103	103
query46	708	762	455	455
query47	740	779	715	715
query48	281	268	184	184
query49	538	229	181	181
query50	728	230	163	163
query51	8150	8032	8066	8032
query52	63	65	56	56
query53	171	181	127	127
query54	213	162	136	136
query55	79	55	55	55
query56	203	177	179	177
query57	696	649	665	649
query58	222	174	153	153
query59	1237	1238	1182	1182
query60	246	190	186	186
query61	149	134	144	134
query62	359	215	185	185
query63	167	145	139	139
query64	2835	803	687	687
query65	1631	1663	1604	1604
query66	1980	267	197	197
query67	10033	9829	9735	9735
query68	2871	906	419	419
query69	327	228	197	197
query70	663	650	612	612
query71	241	171	155	155
query72	2243	1674	1576	1576
query73	609	539	304	304
query74	1976	1222	1116	1116
query75	1149	1095	950	950
query76	2314	650	474	474
query77	251	240	191	191
query78	4088	3724	3252	3252
query79	2710	725	469	469
query80	1631	297	250	250
query81	542	151	134	134
query82	796	122	106	106
query83	274	205	183	183
query84	297	107	91	91
query85	814	314	273	273
query86	473	172	175	172
query87	1001	975	887	887
query88	3154	2046	2029	2029
query89	274	189	161	161
query90	2151	133	122	122
query91	127	112	91	91
query92	99	70	70	70
query93	2636	823	462	462
query94	631	257	224	224
query95	502	243	307	243
query96	790	532	245	245
query97	1057	1076	1021	1021
query98	175	133	131	131
query99	421	344	306	306
Total cold run time: 180250 ms
Total hot run time: 79838 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 14.77 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5b32913a519585871a9bebf50d821551674e67ca, data reload: false

query1	0.01	0.00	0.01
query2	0.07	0.03	0.04
query3	0.25	0.12	0.11
query4	1.60	0.10	0.10
query5	0.17	0.15	0.16
query6	1.30	0.69	0.71
query7	0.03	0.01	0.00
query8	0.05	0.03	0.04
query9	0.28	0.22	0.22
query10	0.35	0.33	0.34
query11	0.16	0.11	0.12
query12	0.14	0.12	0.12
query13	0.29	0.30	0.31
query14	0.45	0.45	0.44
query15	0.37	0.35	0.35
query16	0.23	0.22	0.22
query17	0.64	0.68	0.64
query18	0.16	0.18	0.18
query19	1.23	1.15	1.19
query20	0.02	0.01	0.01
query21	15.50	0.16	0.13
query22	5.04	0.04	0.04
query23	16.18	0.24	0.10
query24	3.01	0.32	0.26
query25	0.12	0.04	0.03
query26	0.79	0.16	0.11
query27	0.04	0.04	0.03
query28	3.62	0.56	0.28
query29	12.48	3.23	2.60
query30	0.26	0.11	0.12
query31	2.74	0.36	0.17
query32	3.54	0.33	0.23
query33	1.39	1.53	1.50
query34	15.37	2.21	1.78
query35	1.77	1.74	1.71
query36	0.45	0.29	0.28
query37	0.06	0.04	0.04
query38	0.05	0.03	0.03
query39	0.03	0.03	0.02
query40	0.11	0.07	0.07
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 90.5 s
Total hot run time: 14.77 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants