feat: state a scroll request as ordering plus position, read tokens from the row - #561
Merged
Conversation
…rom the row A Scrollable names the sort fields in precedence, each in its own direction and in any number, the key that breaks ties with its own direction, the window size, and optionally the row to continue after or before. backward() is gone: descending() orders the key descending and previous() navigates. Every window comes back in the request's sort order, the one reached through previous() included, and hasNext and hasPrevious say whether rows exist after and before the window. The sort and key columns are appended to the select list and read from each row alongside the mapped result, so refs, projections read as another type and custom select types carry navigation tokens; an inline record key keeps reading its value from the mapped record. scrollRef joins pageRef on the repositories. Sort fields must not allow NULL values, checked the way the key is. A cursor string carries the position only, under a fingerprint of the ordering; the size stays with the request, so from(cursor) replaces fromCursor and st.orm.scrollable.maxSize goes away. scroll(int) is slice(int) and returns Slice, which iterates over its content. Order moves out of Pageable so both requests share one sorting vocabulary.
zantvoort
added
enhancement
A cursor comes from a client, so whatever makes it unusable, a malformed string, an earlier format, another ordering or codec registry, or a value of the wrong type, is one condition for the code that receives it: start over from the first window. InvalidCursorException, a PersistenceException, names that condition, where the code used to catch IllegalArgumentException by type and message.
The kit gains a scroll suite: windows over the key in both directions, sort fields with mixed directions, the order and flags of a window reached through previous(), an empty window before the first row, refs and a reference sort field carrying tokens, cursor round trips and an offset slice. The reference sort case reads refs and compares against the raw id and type columns, so no dialect has to parse the pet's date column. Every dialect module runs it the way it runs the other suites.
The documented User has an email, a birth date, a street, a postal code and a city. Examples across the docs, the skills and the class-level Javadoc filtered on an active flag the entity does not have; they now filter on the email domain and the city, group by city and postal code, and delete by a missing postal code.
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
… scrollRef Tampered cursors exercise every refusal of the codec: another format version, trailing bytes, a value count that does not match the ordering, and a null value. The inline record key is scrolled from the record and refused as a sort field and for refs, a delete query is refused, the projection repositories scroll refs in all three modules, and a scroll inside a SQL log scope reports its rows. Order.reversed() had no caller and goes.
…t page Page.nextPageable() and previousPageable() were Window.next() and previous() under other names, and Pageable.previous() on page 0 answered with itself, which lets a loop step backwards forever. Both previous() methods now return null on the first page, the way Window.previous() is null when nothing precedes.
The row-plus-cursor pair only ever comes out of KeyedQuery, so it lives there as KeyedQuery.Row rather than in a file of its own.
Position no longer exposes the row values it names. Its only public member says on which side of the row the request continues; the values are read by the engine through st.orm.impl.PositionAccess, a package storm-foundation exports to storm-core alone, the same way the engine's own impl packages are reachable by Storm's modules only. CursorFactory deals in the side and the values rather than in the foundation type, and the cursor bridge rebuilds the position on the foundation side. The stale Javadoc links to the removed Scrollable#fromCursor now point at from(String).
SqlOperation linked to SqlTemplate, which lives in storm-core and is not visible from the foundation; the name is now plain code. In storm-core the EntityRepository class comment starts its headings at the second level, below the implicit heading of the class name, Templates closes its paragraphs with the right tag, and TransactionScope links to open and complete by their full signatures.
slice takes the same Pageable as page, reads one row beyond the page size to decide hasNext, derives hasPrevious from the page number, and navigates with next() and previous(). It joins page on the query builder with the same overloads, and slice and sliceRef join page and pageRef on the entity and projection repositories in core, Java 21 and Kotlin. Slice is a record like Page; Page, Slice and Window each iterate over their content, so the shared interface and the builder-offset slice(int) are gone. Scrollable drops at(Position), ascending() and hasPosition(): a window hands back the request that continues from it, ascending is the default, and a request without a position has a null position. The keyed-result hook of the query builders is package-private. Its signature names a package-private type, so no subclass outside the package could implement it, and the wider access only drew an IDE warning. The pagination docs pick the read from one table that sets slice, page, scroll and windows side by side, and the skills carry a compact copy.
Window, Page and Slice each iterate on their own, so the copy that called a window a Slice says it iterates. The Window flags say whether rows existed after and before the window in sort order. The pagination tutorial navigates with next(), filters on the documented city field and lists slice beside scroll as the reads without a count. The glossary gains Position and Slice, and the common patterns page points at the read comparison table.
Position is an interface in the foundation with after() as its only member. The engine's PositionImpl carries the row values; Scrollable.after and before ask storm-core to build it through CursorHelper, the way toCursor and from already ask it to encode and decode a cursor, and the engine reads its own record when it builds the window or the cursor. The foundation has no impl package and no qualified export. A position the engine did not build, and a position whose value count does not match the ordering, are refused at scroll time. The Scrollable and Position tests live in storm-core, since building a position needs the engine, the way the cursor tests already do.
Each foundation bridge covers one concern and points at one core class: PositionHelper reaches PositionFactory the way CursorHelper reaches CursorFactory, and the cursor pair is back to cursors only. The native-image configuration registers the new pair under the same condition as the others.
Slice is the interface a Page, a Window and a plain slice implement: content, the two flags, and iteration over the content. slice(pageable) returns that shape, backed by a package-private record, and a plain slice navigates through the Pageable that produced it, since next() on the shared interface would have to return two kinds of request. Page and Window inherit the iteration instead of carrying their own copy.
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
| return (Position) POSITION_METHOD.invoke(null, values, after); | ||
| } catch (InvocationTargetException e) { | ||
| throw e.getTargetException(); | ||
| } catch (ReflectiveOperationException e) { |
Every dialect now renders the offset reads as well as the keyset ones: a full page runs the count and reports the total, a short last page derives the total from itself, a page beyond the end is empty and still counted, a ref page carries the same total, and a total the application holds skips the count query. The ref slice states its ordering, since SQL Server refuses an OFFSET without an ORDER BY.
… with it The unreleased section opens with the one fix of the release and presents the scrolling, cursor and offset-read changes as the adjustments that let windows take a stream's place, in the order a reader meets them.
The suite checks page, slice, scroll and windows on every dialect, so its name follows the docs page that covers them, and the seven dialect wrappers follow the suite.
…d limit stay the baseline below
SQL Server accepts OFFSET and OFFSET-FETCH only after an ORDER BY, so an unordered first page, the documented page(0, 20), failed there while it ran on every other database. The dialect now names the ordering an unordered offset needs, a constant ORDER BY (SELECT NULL) on SQL Server and none elsewhere, and the select builder renders it when a query applies an offset without an ordering of its own. The pagination conformance suite reads an unordered page, slice, ref slice and offset on every dialect.
A NULL never compares, so a row whose sort or key value is NULL would fall out of every window rather than appear in one of them; Storm refuses the query instead of skipping the row silently. The scrolling page and the entity, query and repository skills state it where the rule is given.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #560
Keyset scrolling kept the right shape and broke four promises its types made. This states ordering and position as two things, reads the navigation tokens from the row so they exist for every result type, allows any number of sort fields, and makes windows iterable.
Request
Scrollableis an ordering, a size and optionally a position: the sort fields in precedence, each with its own direction; the key with its own direction as the tiebreaker; and the row to continue after or before.Ordermoves out ofPageabletost.orm.Orderso both requests share one sorting vocabulary.backward()is gone:descending()orders the key descending, andprevious()navigates.Sort fields must not allow NULL values, checked the way the key is, since
WHERE field > ?drops NULL rows silently.Window
Every window is in the request's sort order. A window before a row is read in the reversed ordering and turned around, so
previous()reads exactly likenext().hasNextandhasPrevioussay whether rows exist after and before the window in sort order: the anchor row of a position lies on the side the request continued from, the other side is decided by the extra row fetched.The sort and key columns are appended to the select list and read from each row alongside the mapped result, in
QueryImpl.readKeyedRows, so refs, projections read as another type and custom select types carry tokens. An inline record key spans several columns and is compared as a whole, so it keeps reading its value from the mapped record, which needs the entity type as the result;windows(size)therefore iterates a compound primary key too.scrollRefjoinspageRefon both repositories.Pagenavigates the wayWindowdoes:next()andprevious()replacenextPageable()andpreviousPageable(), andprevious()is null on the first page, onPageandPageablealike, where the request used to answer with itself.SliceextendsIterablewithsize(),isEmpty()andstream().scroll(int)isslice(int)and returnsSlice, the shape without a key: the query's own ordering and offset,hasNextfrom one extra row,hasPreviousfrom the offset.Cursor
The cursor string is a position: whether to continue after or before a row, and that row's values, under a fingerprint of the ordering and the codec registry. The size stays with the request, so
Scrollable.of(key, size)...from(cursor)replacesfromCursor, a client may change the page size between requests, andst.orm.scrollable.maxSizegoes away. Format version 2; cursors from 1.14.0 are refused by the version check.A refused cursor throws
InvalidCursorException, aPersistenceException, whatever the reason: malformed, earlier format, another ordering or codec registry, wrong value type. A cursor comes from a client, so a web layer maps this one exception to its "start over" response.Docs
The scrolling and cursors pages are rewritten around ordering plus position; the "Window Type Parameters" section is gone because the case it explained no longer exists. The repositories, batch-streaming, glossary and configuration pages, the pagination tutorial and the four query and repository skills follow. Every example in docs, skills and class-level Javadoc now uses the documented
Userfields: filters onemailandcity, noactiveflag.Tests
Keyset scrolling joins the Technology Compatibility Kit:
AbstractPaginationConformanceTestruns on all seven dialects, covering windows over the key in both directions, mixed-direction sort fields, previous-window order and flags, an empty window before the first row, refs and a reference sort field carrying tokens, cursor round trips and an offset slice.KeysetScrollIntegrationTestcovers mixed-direction sort fields, previous-window order and flags, an empty window before the first row, refs and a custom grouped select type carrying tokens, a reference field as sort field, the nullable sort field refusal,windowsrefusing a before position,slicewith an offset, and window iteration. The existing scroll suites in core, Java and Kotlin are migrated to the new factories with their before-window expectations inverted. Cursor serialization tests cover two sort fields, the size not being part of the cursor, and refusal on a mismatched key, sort, direction or value type.A slice is a page without the count query
slice(pageable)takes the samePageableaspage, reads one row beyond the page size to decidehasNext, deriveshasPreviousfrom the page number, and navigates through the request'snext()andprevious(). It has the same overloads aspageon the query builder, andsliceandsliceRefjoinpageandpageRefon the entity and projection repositories in core, Java 21 and Kotlin.Slicestays the interfacePageandWindowimplement, and now iterates over its content; the old builder-offsetscroll(int)is gone. The pagination docs pick the read from one table that sets slice, page, scroll and windows side by side.Position is opaque
Positionsays on which side of the row the request continues, and nothing else, like the cursor string that carries it. It is an interface in the foundation; the engine builds the implementation that carries the row values, through a bridge of its own beside the cursor's, and reads it back when it builds the window. AScrollablestates its position throughafter,beforeandfrom, a window hands back the request that continues from it, and a request without a position has a null position.An offset without an ordering reads on SQL Server
The conformance run surfaced it: SQL Server accepts
OFFSETandOFFSET-FETCHonly after anORDER BY, so the documented unordered first page,page(0, 20), failed there while it ran on every other database.SqlDialect.orderByForOffset()names the ordering an unordered offset needs,ORDER BY (SELECT NULL)on SQL Server and none elsewhere, and the select builder renders it when a query applies an offset without an ordering of its own. The pagination suite reads an unordered page, slice, ref slice and offset on all seven dialects.