Support limiting prefixes probed for MySQL to constrain runtime - #38164
Merged
Conversation
peterdukelarsen
changed the title
peterdukelarsen
force-pushed
the
pl/mysql-partition-request-budget
branch
from
August 11, 2026 17:26
f3a2461 to
90dc1b3
Compare
peterdukelarsen
force-pushed
the
pl/mysql-partition-request-budget
branch
from
August 11, 2026 17:38
90dc1b3 to
f3a000c
Compare
peterdukelarsen
force-pushed
the
pl/mysql-partition-request-budget
branch
8 times, most recently
from
August 11, 2026 19:40
48cfbb4 to
f071ccd
Compare
peterdukelarsen
changed the title
peterdukelarsen
marked this pull request as ready for review
August 11, 2026 19:48
peterdukelarsen
force-pushed
the
pl/mysql-partition-request-budget
branch
from
August 11, 2026 20:15
f071ccd to
1d68e03
Compare
peterdukelarsen
force-pushed
the
pl/mysql-partition-request-budget
branch
from
August 11, 2026 22:41
1d68e03 to
ecefb45
Compare
peterdukelarsen
force-pushed
the
pl/mysql-partition-request-budget
branch
from
August 11, 2026 23:02
ecefb45 to
f5d757b
Compare
peterdukelarsen
force-pushed
the
pl/mysql-partition-request-budget
branch
from
August 11, 2026 23:35
f5d757b to
9304f11
Compare
peterdukelarsen
force-pushed
the
pl/mysql-partition-request-budget
branch
from
August 11, 2026 23:42
9304f11 to
bcc67c6
Compare
peterdukelarsen
force-pushed
the
pl/mysql-partition-request-budget
branch
from
August 12, 2026 00:12
bcc67c6 to
1b0861e
Compare
peterdukelarsen
force-pushed
the
pl/mysql-partition-request-budget
branch
2 times, most recently
from
August 12, 2026 01:23
a097f1c to
c33afc9
Compare
ublubu
approved these changes
Aug 17, 2026
ublubu
left a comment
ublubu
left a comment
Contributor
There was a problem hiding this comment.
Looks like it bounds the probing and tests some of the cases it guards against. 👍
| /// cover them. | ||
| struct MockDb { | ||
| keys: Vec<String>, | ||
| requests: usize, |
Contributor
There was a problem hiding this comment.
What sort of requests is this counting? Asking because some operations cost 1, while others cost 2.
Contributor
Author
There was a problem hiding this comment.
The 1 and 2 here map to the number of underlying requests made to mysql. In the actual implementation one method makes two calls while the other makes one so I wanted to make sure the number of individual requests was bounded in about the way I expected (less than 4 requests per prefix).
| let budget = 20; | ||
| let boundaries = partition(&mut db, 16, count, 10, budget).await?; | ||
| assert!( | ||
| db.requests <= 80, // 4x budget |
Contributor
There was a problem hiding this comment.
Limiting the probes to 20 prefixes makes sense. Where does the 4x upper bound come from?
Contributor
Author
There was a problem hiding this comment.
We make up to 4 requests per prefix -- one to get the first key matching the current prefix (only once per parent amortized across children) , two to get the next, and then one to estimate the size.
peterdukelarsen
force-pushed
the
pl/mysql-partition-request-budget
branch
from
August 17, 2026 18:13
c33afc9 to
e54c103
Compare
peterdukelarsen
force-pushed
the
pl/mysql-partition-request-budget
branch
from
August 17, 2026 18:48
e54c103 to
aff95e4
Compare
Bound the prefix partitioner to a per-table probe query budget scaled to the estimated row count, 2500 requests per billion rows by default via the mysql_source_snapshot_partition_requests_per_billion_rows dyncfg, with a floor of 256 so small tables can afford their handful of splits. Probes run sequentially, so this caps the wall-clock that partitioning can add to snapshot start in proportion to the snapshot work it optimizes. An exhausted budget stops splitting early and leaves coarser buckets, never incorrect ones. Adds a mock test asserting the partitioner never exceeds its budget and still returns a valid ordered boundary list when truncated.
peterdukelarsen
force-pushed
the
pl/mysql-partition-request-budget
branch
from
August 17, 2026 19:02
aff95e4 to
4384c69
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Part of SS-97.
In degenerate cases with high cardinality of first characters, many short prefixes (i.e. "a", "aa", "aaa", "aaaa"...), or unexpected bugs, we want to ensure partitioning terminates in a reasonable amount of time.
Description
Bounds the partitioner's probe traffic with a request budget derived from the table's estimated row count, via the new
mysql_source_snapshot_partition_requests_per_billion_rowsdyncfg (floored at 256 requests). When the budget runs out the walk stops splitting and emits coarser partitions, so cost is capped without losing correctness. The budget also serves as the hard bound on the walk for degenerate key spaces.Adds mock tests for budget-bounded probing and for termination on non-advancing prefixes.
🤖 Generated with Claude Code