Skip to content

inspect and inspect_stamped do not observe frontiers - #834

Merged
frankmcsherry merged 2 commits into
TimelyDataflow:masterfrom
frankmcsherry:inspect-frontier-interest
Nov 8, 2026
Merged

frankmcsherry merged 2 commits into
TimelyDataflow:masterfrom
frankmcsherry:inspect-frontier-interest

Conversation

@frankmcsherry

@frankmcsherry frankmcsherry commented Nov 8, 2026 •

Copy link
Copy Markdown
Member

inspect no longer observes frontiers, so it no longer keeps an enclosing region awake, and a new inspect_stamped does the same for stamped containers.

inspect was built on inspect_core, which is scheduled whenever its input frontier changes.
A region takes the strongest frontier interest among its children, so one inspect inside a region means the region is scheduled every time time moves, and then does progress work for all of its children.
This PR adds inspect_stamped(|stamp, container| ..), built on unary, which declares no frontier interest, so it is scheduled only when containers arrive.
inspect is built on inspect_stamped.
inspect_core still observes frontiers, and its documentation now says what that costs; it was previously the only way to see stamps.

Data.
N dataflows of input -> region { map^L -> inspect } -> probe, one record into one input per round, all inputs advanced, one worker, release build.
On master inspect is inspect_core underneath, so the left column is master's behavior.

N x L inspect_core (master's inspect) inspect (this PR)
100 x 100 2,226 µs/round 184 µs/round
1,000 x 100 24,772 µs/round 1,444 µs/round
100 x 1,000 24,832 µs/round 942 µs/round

This came up in a spreadsheet engine built on differential dataflow (see #833), which observes each formula's output with inspect.
With its 2,000 formulas grouped into regions, an edit that reaches one formula took 1.75 ms per edit with frontier-observing inspects, and 0.07 ms with this change.

The new test counts how often a region holding only an inspect operator is scheduled as an input advances 100 rounds without data: 104 times with inspect_core, and 5 times with either inspect_stamped or inspect.

🤖 Generated with Claude Code

`inspect` was built on `inspect_core`, which is scheduled whenever its input frontier changes.
A region takes the strongest frontier interest of its children, so one `inspect` kept its whole region awake, and the region did progress work for every child each time any frontier moved.

`inspect` now builds on `unary`, which declares no frontier interest.
It is scheduled only when records arrive.
`inspect_core` still observes frontiers, and its documentation says what that costs.

A new test counts how often a region holding only an `inspect` is scheduled as an input advances without data.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
`inspect_stamped` runs a closure on each container and the stamp under which it travels.
Like `inspect`, it does not observe frontiers, so it is scheduled only when containers arrive.
Before this, the only way to see stamps was `inspect_core`, which also observes frontiers and is scheduled whenever they change.

`inspect` is now built on `inspect_stamped`, so there is one data-only implementation.
The test checks that a region holding either one sleeps while frontiers move.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@frankmcsherry frankmcsherry changed the title inspect does not observe frontiers inspect and inspect_stamped do not observe frontiers Nov 8, 2026
antiguru

This comment was marked as duplicate.

@antiguru antiguru left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, thank you!

@frankmcsherry
frankmcsherry merged commit 33d29ed into TimelyDataflow:master Nov 8, 2026
9 checks passed
@frankmcsherry
frankmcsherry deleted the inspect-frontier-interest branch November 8, 2026 13:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants