Skip to content

Guard the CHASM pure-task invalidation invariant for scheduler tasks - #11432

Draft
davidporter-id-au wants to merge 10 commits into
temporalio:mainfrom
davidporter-id-au:chasm-enforce-pure-task-invalidation
Draft

davidporter-id-au wants to merge 10 commits into
temporalio:mainfrom
davidporter-id-au:chasm-enforce-pure-task-invalidation

Conversation

@davidporter-id-au

@davidporter-id-au davidporter-id-au commented Aug 6, 2026 •

Copy link
Copy Markdown
Contributor

What changed?

  • Add chasmtest.FirePureTasksStrict, which enforces the framework assumption that a pure task must become invalid once it has executed. It re-runs each task immediately after execution: a compliant task fails its own validator and does nothing, a violator executes a second time and is reported.
  • Cover all four scheduler pure tasks with it: generate, backfill, the delayed form of processBuffer, and idle.

No production code changes — tests and a test helper only.

Why?

A validator returning false is the only mechanism that prunes a task from ComponentAttributes.PureTasks (closeTransactionCleanupInvalidTasks). A task that stays valid is never pruned, remains past-due, and so stays the tree's earliest pure task. Because only the earliest task is ever considered for a physical timer, and its PhysicalTaskStatus is already created, closeTransactionGeneratePhysicalPureTask takes its early return and creates nothing.

The result is that the entire execution loses pure-task processing, not just the offending component. For a schedule that means the Generator, the Invoker's ProcessBuffer, the idle task and every Backfiller stop firing simultaneously — silently, with no failed task, no DLQ entry and no retry counter, since the physical timer succeeded. There is no ordinary self-healing path: an immediate task executes inline and is never persisted, and a future-dated task sorts behind the stranded one. Recovery requires RefreshTasks, replication recreating the node, or a task backdated earlier than the stranded one.

This is the check described by the existing TODO in chasm.Node.EachPureTask ("Add a validation for that and return an internal error if tasks is still valid after processing"), applied from the test side. If the framework grows that assertion itself, these tests can fall back to FirePureTasks.

How did you test it?

  • added new unit test(s)

Mutation-tested rather than merely observed green: making GeneratorTaskHandler.Validate return true, nil fails three of the four tests with pure task invariant violated: *schedulerpb.GeneratorTask ... executed a second time. Each test also asserts that its target task type was actually exercised, so none can pass vacuously — that assertion is what caught processBuffer being uncovered initially.

Scope and known gaps

  • Only persisted, timer-driven pure tasks are observed. Immediate pure tasks execute inline during CloseTransaction and are never persisted, so they cannot strand a timer and are structurally out of scope. This is why processBuffer is covered via its delayed (nextBackoffDeadline) form, and why a trigger-immediately Backfiller is not covered — its task is consumed inline within the PatchSchedule transaction.
  • Side-effect tasks are out of scope: each gets its own physical task, so they have no single-timer bottleneck.
  • *persistence.ChasmVisibilityTaskData (CHASM core library) is a pure task present in every scheduler's tree and shares the execution's single timer, but it belongs to the core library rather than the scheduler and is not covered here.

chaptersix and others added 8 commits August 4, 2026 10:54
LastProcessedTime is a schedule-range cursor and cannot identify stale task deliveries. Stamp Backfiller tasks with a generation so superseded and redelivered tasks are rejected, while preserving generation-zero continuations during rolling upgrades.
CHASM relies on an unenforced assumption: a pure task MUST become invalid
once it has executed. A validator returning false is the only mechanism
that prunes a task from ComponentAttributes.PureTasks
(closeTransactionCleanupInvalidTasks). A task that stays valid is never
pruned, remains past-due, and therefore stays the tree's earliest pure
task -- and because only the earliest task is ever considered for a
physical timer, and its PhysicalTaskStatus is already `created`,
closeTransactionGeneratePhysicalPureTask declines to create another one.

The whole execution then loses pure-task processing, not just the
offending component. For a schedule that means the Generator, the
Invoker's ProcessBuffer, the idle task and every Backfiller stop firing
at once, silently, with no ordinary self-healing path -- recovery needs
RefreshTasks, replication recreating the node, or a task backdated
earlier than the stranded one.

The Backfiller violated this before the preceding change: it validated a
wall-clock task ScheduledTime against LastProcessedTime, which for a
Backfiller is a position in the requested backfill *range*. For a
historical range that comparison never returned false.

Add chasmtest.FirePureTasksStrict, which re-runs each pure task
immediately after it executes -- a compliant task fails its own
validator and does nothing, a violator executes twice -- and use it to
cover all four scheduler pure tasks: generate, backfill, the delayed
form of processBuffer, and idle. Each test also asserts that its target
task type was actually exercised, so it cannot pass vacuously.

This is the check described by the existing TODO in
chasm.Node.EachPureTask, applied from the test side. Once the framework
enforces the invariant itself these tests can fall back to
FirePureTasks.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants