Skip to content

Upgrade to Storm 1.14.1 and publish the 2026-09-13 run - #9

Merged
zantvoort merged 2 commits into
mainfrom
bench/storm-1141-upgrade
Sep 14, 2026
Merged

zantvoort merged 2 commits into
mainfrom
bench/storm-1141-upgrade

Conversation

@zantvoort

@zantvoort zantvoort commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

Storm 1.14.1 states a scroll request as an ordering, a size and a position, so the Scrollable.of(key, lastValue, size) factory the keyset workload used is gone. The workload is unchanged — one page of 20 rows seeking past a cursor — and now builds its request the way 1.14.1 does:

pets.scroll(Scrollable.of(Pet_.id, Dataset.PAGE_SIZE).after(cursor)).content

What the statements do

Comparing printSql across the two versions, every workload emits the same SQL except the keyset query, whose select list now carries the key column a second time: 1.14.1 reads a window's position out of the row rather than from the request, so the sort and key columns join the selection. One extra column over a 20-row page, on the one workload that scrolls.

The run

Full field on one dedicated instance (4 vCPU / 16 GB, Ubuntu 24.04.5, AMD EPYC 9V74), Storm built from v1.14.1 at 15f63804: workflow run. Committed under results/2026-09-13/ with its metadata and mirrored into results/summary.md.

The instance is not the one the 2026-09-03 table came from. The JDBC baseline measures 101.9 µs/op against 140.7 µs/op there, −28% on a different CPU model, and the drift guard flagged it:

JDBC baseline moved -28% vs results/2026-09-03/ (140.7 -> 101.9 µs/op) — different-speed hardware. Do NOT compare this run's absolutes to the published table; compare within-run (relative to JDBC) or run a same-instance A/B.

So the two tables are not cell-comparable, and each score reads against the JDBC baseline inside its own table.

Where Storm lands within this run

Fastest library on nine of the twelve workloads: the primary-key lookup, the projection, keyset pagination, the hundred- and thousand-row joins, the batch insert, the single-row update, the create-then-amend transaction and the object graph's runner-up place are led or shared by Storm. Three go elsewhere, and they are worth separating by how solid the margin is within this run, comparing each gap against the two cells' combined JMH error:

Workload Leader Storm Gap Error band Reading
graphInsert Ktorm 2418.6 2450.7 1.3% 1.7% inside the band: a tie
dynamic Hibernate 924.4 936.8 1.3% 1.1% just outside: Hibernate ahead
joinWithMapping10 Jimmer 638.8 659.5 3.2% 1.9% outside: Jimmer ahead

jOOQ keeps the object graph outright, 845.5 against 1152.7, a 36% gap far beyond any error band, with its MULTISET JSON aggregate against repeated join rows.

Those orderings are within-run. The reproducibility repeat committed under results/2026-09-03-repeat/ measured individual scores moving about 1% between executions on equivalent hardware, worst case 6.5%, which is enough to reorder a group this close, so none of the three should be read as a stable ranking across runs.

Data quality

Storm's twelve cells are the cleanest in the run: JMH error at most 2.2%, fork spread at most 7.6%. Nine of the 97 cells are noisier than the rest and should not be quoted as exact figures — updateById on Exposed DAO (56% fork spread, ±14.4%), multiStatement on Hibernate (43%, ±14.3%), joinWithMapping100 on jOOQ (17%, ±7.7%), batchInsert on Ktorm (17%), objectGraph and updateById on Hibernate (17% and 13%), and the JDBC baseline row itself (15%). The rankings around them are unaffected; the gaps there are far wider than the noise.

Ratios are comparable across runs even when absolutes are not: Storm's score against the JDBC baseline moved at most 4.3% on any workload against 2026-09-03, most under 2%, so this run agrees with the previous one. The largest move is keyset at +3.9%, which is also the one workload whose SQL changed — consistent with the extra key column, though within the repeat study's spread rather than proof of it.

Nothing here is a before/after against 1.14.0: different hardware. The baseline_ref A/B mode on one instance is the instrument for that, and it is worth running for the per-statement stream guard and the read-only write check that 1.14.1 adds to the hot path.

Storm 1.14.1 states a scroll request as an ordering, a size and a
position: the of(key, lastValue, size) factory is gone, so the keyset
workload and the printSql dump build their request with
Scrollable.of(Pet_.id, PAGE_SIZE).after(cursor). The workload is
unchanged, one page of 20 rows seeking past a cursor.

The statement it runs is not quite unchanged. Comparing the printSql
output across the two versions, every workload emits the same SQL except
the keyset query, whose select list now carries the key column a second
time: 1.14.1 reads a window's position out of the row rather than from
the request, so the sort and key columns join the selection. One extra
column over a 20-row page, on the one workload that scrolls.
The full field on one dedicated instance, Storm built from v1.14.1 at
15f63804. Committed under results/2026-09-13 with its metadata, and
mirrored into results/summary.md as the current run.

The instance is not the one the 2026-09-03 table came from: the JDBC
baseline measures 101.9 µs/op against 140.7 there, -28%, on a different
CPU model, and the drift guard flagged it. The two tables are not
cell-comparable, so each score reads against the JDBC baseline inside its
own table, as the standing rule says.

Within this run Storm is the fastest library on nine of the twelve
workloads and second on the other three, behind Hibernate on the dynamic
query by 1.3%, Jimmer on the ten-row join by 3.2%, and Ktorm on the graph
insert by 1.3%. jOOQ keeps the object graph with its MULTISET aggregate,
36% ahead. The keyset query carries one extra column since 1.14.1 reads a
window's position out of the row, which is the only SQL that changed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant