Skip to content

[fix](http) Handle metadata pagination boundaries - #67816

Open
morrySnow wants to merge 1 commit into
apache:masterfrom
morrySnow:fix/meta-info-pagination
Open

[fix](http) Handle metadata pagination boundaries#67816
morrySnow wants to merge 1 commit into
apache:masterfrom
morrySnow:fix/meta-info-pagination

Conversation

@morrySnow

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Metadata REST endpoints can mishandle large pagination values in three ways:

  • An offset beyond the available list is clamped to the final item instead of producing an empty page.
  • Adding large limit and offset integers can overflow and create an invalid sub-list range.
  • Values outside the integer range raise parsing exceptions that are reported as internal errors.

The behavior affects both the legacy metadata routes and the v2 metadata routes. It can be reproduced by requesting a metadata list with a very large offset, with both limit and offset set to the maximum integer, or with a pagination value larger than the supported numeric range.

What is the root cause?

Both controllers contain duplicate pagination logic based on int parsing and direct limit-plus-offset arithmetic. The lower bound is also capped at size minus one, which turns a past-the-end offset into the last element.

What changes are included?

  • Add one shared pagination helper for both controller versions.
  • Parse non-negative values as long integers and map malformed or unsupported values to a stable bad-request response.
  • Return the half-open empty range at the list size when the offset reaches or exceeds the end.
  • Compute the upper bound from the remaining item count, avoiding overflow-prone addition.
  • Add focused unit tests and HTTP regression coverage for both route versions.

Release note

Fix metadata REST pagination for large and invalid limit or offset values.

Check List (For Author)

  • Test: Unit Test and Regression Test
  • Behavior changed: Yes. Out-of-range offsets now return an empty page, and invalid pagination values return a stable bad-request response.
  • Does this need documentation: No

### What problem does this PR solve?

Problem Summary:
Metadata REST endpoints could return the last item for an offset beyond the list, overflow when adding large limit and offset values, or expose numeric parsing failures as internal errors.

This change makes the legacy and v2 endpoints share an overflow-safe pagination helper. It parses non-negative long values, returns an empty range when the offset is at or beyond the list end, bounds the page by the remaining item count without unsafe addition, and reports invalid values as bad requests. Unit and HTTP regression tests cover normal, out-of-range, overflow, and malformed inputs.

### Release note

Fix metadata REST pagination for large and invalid limit or offset values.

### Check List (For Author)

- Test: Unit Test and Regression Test
- Behavior changed: Yes. Out-of-range offsets now return an empty page, and invalid pagination values return a stable bad-request response.
- Does this need documentation: No
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@morrySnow

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 88.46% (23/26) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 16728 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 17d5b814bb0240776ddc2fce97495998f9ce21f7, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17617	3059	3104	3059
q2	2063	261	216	216
q3	10258	908	513	513
q4	4667	253	203	203
q5	7677	571	382	382
q6	136	116	93	93
q7	518	496	393	393
q8	9240	884	875	875
q9	3373	2377	2390	2377
q10	6505	855	715	715
q11	395	202	180	180
q12	616	258	198	198
q13	18146	1527	1160	1160
q14	156	151	140	140
q15	q16	438	398	370	370
q17	1379	886	802	802
q18	3130	2227	2202	2202
q19	1273	829	775	775
q20	369	283	196	196
q21	5654	1655	1897	1655
q22	327	263	224	224
Total cold run time: 93937 ms
Total hot run time: 16728 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3433	3384	3397	3384
q2	499	391	382	382
q3	2245	2274	2165	2165
q4	1184	1166	885	885
q5	2151	2095	2099	2095
q6	164	120	88	88
q7	1034	904	837	837
q8	1596	1396	1417	1396
q9	3092	3084	3069	3069
q10	1867	1806	1644	1644
q11	355	268	253	253
q12	454	430	341	341
q13	1503	1550	1136	1136
q14	170	165	157	157
q15	q16	389	403	360	360
q17	3614	3287	3195	3195
q18	4767	4435	4752	4435
q19	843	799	955	799
q20	1012	979	860	860
q21	3872	3115	3294	3115
q22	405	338	324	324
Total cold run time: 34649 ms
Total hot run time: 30920 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 81470 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 17d5b814bb0240776ddc2fce97495998f9ce21f7, data reload: false

query5	4239	407	347	347
query6	367	136	122	122
query7	4962	426	226	226
query8	287	123	115	115
query9	8692	2880	2893	2880
query10	417	235	187	187
query11	5378	1057	910	910
query12	119	73	70	70
query13	1193	437	328	328
query14	6094	2206	2088	2088
query14_1	1985	1987	1997	1987
query15	171	120	106	106
query16	916	378	365	365
query17	808	464	374	374
query18	2330	334	234	234
query19	161	138	108	108
query20	71	72	72	72
query21	205	101	88	88
query22	5509	5436	5298	5298
query23	6935	6208	5951	5951
query23_1	6065	6020	6198	6020
query24	7269	1106	777	777
query24_1	805	787	768	768
query25	420	316	228	228
query26	1219	244	128	128
query27	2781	386	248	248
query28	4716	1468	1499	1468
query29	916	403	353	353
query30	259	158	129	129
query31	811	401	331	331
query32	128	69	71	69
query33	443	212	178	178
query34	992	813	469	469
query35	396	403	344	344
query36	567	564	512	512
query37	118	77	66	66
query38	993	842	804	804
query39	488	498	515	498
query39_1	450	461	450	450
query40	207	89	75	75
query41	53	51	53	51
query42	71	71	72	71
query43	240	241	207	207
query44	995	528	535	528
query45	112	104	113	104
query46	735	814	525	525
query47	761	728	693	693
query48	315	312	226	226
query49	544	235	181	181
query50	705	263	195	195
query51	8089	7874	7872	7872
query52	66	69	62	62
query53	194	197	142	142
query54	214	181	145	145
query55	70	62	56	56
query56	189	169	169	169
query57	691	764	674	674
query58	205	165	153	153
query59	1235	1216	1096	1096
query60	222	176	165	165
query61	127	119	124	119
query62	370	199	176	176
query63	165	137	139	137
query64	2667	674	642	642
query65	1601	1563	1560	1560
query66	1934	259	196	196
query67	9601	9952	9688	9688
query68	2994	1203	712	712
query69	357	216	196	196
query70	689	635	622	622
query71	266	186	163	163
query72	2295	1896	1526	1526
query73	661	592	354	354
query74	2003	1234	1143	1143
query75	1170	1091	952	952
query76	2366	709	528	528
query77	257	264	214	214
query78	3834	3679	3199	3199
query79	1163	865	571	571
query80	1258	315	269	269
query81	506	160	130	130
query82	639	121	94	94
query83	320	212	191	191
query84	291	111	89	89
query85	914	329	276	276
query86	396	177	170	170
query87	1013	968	884	884
query88	2774	2085	2105	2085
query89	291	190	176	176
query90	1902	137	126	126
query91	127	120	97	97
query92	84	73	68	68
query93	1325	1088	682	682
query94	654	244	220	220
query95	516	250	237	237
query96	838	614	283	283
query97	1085	1057	1015	1015
query98	155	137	132	132
query99	415	350	305	305
Total cold run time: 176034 ms
Total hot run time: 81470 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 14.65 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 17d5b814bb0240776ddc2fce97495998f9ce21f7, data reload: false

query1	0.00	0.00	0.01
query2	0.07	0.04	0.03
query3	0.26	0.10	0.10
query4	1.60	0.09	0.10
query5	0.17	0.16	0.17
query6	1.27	0.72	0.67
query7	0.03	0.00	0.00
query8	0.04	0.03	0.03
query9	0.28	0.21	0.22
query10	0.34	0.35	0.34
query11	0.16	0.11	0.12
query12	0.14	0.13	0.12
query13	0.30	0.31	0.30
query14	0.45	0.45	0.46
query15	0.36	0.35	0.35
query16	0.22	0.23	0.20
query17	0.71	0.68	0.71
query18	0.17	0.18	0.16
query19	1.13	1.16	1.15
query20	0.01	0.01	0.01
query21	15.45	0.16	0.12
query22	5.05	0.04	0.04
query23	16.17	0.26	0.10
query24	3.02	0.29	0.23
query25	0.11	0.05	0.03
query26	0.80	0.16	0.11
query27	0.03	0.03	0.03
query28	3.66	0.51	0.27
query29	12.53	3.17	2.57
query30	0.26	0.12	0.12
query31	2.76	0.38	0.17
query32	3.52	0.32	0.24
query33	1.38	1.54	1.46
query34	15.37	2.20	1.75
query35	1.73	1.74	1.73
query36	0.45	0.30	0.30
query37	0.06	0.04	0.04
query38	0.04	0.02	0.02
query39	0.03	0.03	0.02
query40	0.12	0.08	0.08
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.02	0.03
Total cold run time: 90.39 s
Total hot run time: 14.65 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants