Queue drain stalled 26 hours behind a run at `waiting`, which the watchdog cannot cancel

[chrikrah]

What happened

Pull request dashboard queue drain completed nothing for about 26 hours, between 2026-09-12T20:26:24Z and
2026-09-13T22:24:43Z. It has recovered, so this reports a closed window.

Last success before the gap	run 34717092723, 2026-09-12T20:26:24Z
Run that entered waiting	run 34717262220, created 2026-09-12T20:29:56Z, last updated 20:30:23Z
Dispatches cancelled during the window	25, first 2026-09-12T20:29:56Z, last 2026-09-13T21:24:07Z, one per hour
Recovery	run 34786711098, 2026-09-13T22:24:43Z, success
34717262220 cancelled	2026-09-13T22:27:54Z, three minutes after the recovery

The 25 cancellations look like GitHub displacing the pending run rather than the watchdog acting. Each one
is cancelled two or three seconds after its own successor is created, which a poller on a 15 minute cycle
would not reproduce:

run 34720120762 created 2026-09-12T21:29:58Z cancelled 2026-09-12T22:30:37Z | successor created 22:30:35Z
run 34722944867 created 2026-09-12T22:30:35Z cancelled 2026-09-12T23:31:07Z | successor created 23:31:05Z
run 34725619681 created 2026-09-12T23:31:05Z cancelled 2026-09-13T00:31:08Z | successor created 00:31:06Z

pull-request-dashboard-drain.yml sets concurrency: group: pull-request-dashboard-queue-drain with
cancel-in-progress: false, so one run may hold the group while a single successor waits behind it. That
matches the pattern above.

Why nothing cleared the head run

workflow-watchdog.mjs:14 cancels only runs whose status is in BLOCKING_RUN_STATUSES:

const BLOCKING_RUN_STATUSES = new Set(["in_progress", "queued"]);
// GitHub reports concurrency-held runs as "pending" or "waiting", and a run
// admitted to the group but waiting for a runner as "queued".
const WAITING_RUN_STATUSES = new Set(["queued", "pending", "waiting"]);

waiting is in the second set and not the first, and the filter at :46 tests the first. So a run parked at
waiting is never a cancellation candidate, and by the comment's own reading waiting is one of the two
statuses a concurrency-held run reports. Run 34717262220 sat there for 26 hours while the watchdog had no
candidate to act on.

A second gap, and it is not the same one

Two pull-request-dashboard.yml runs have been at waiting since 2026-08-29T19:42:54Z and
2026-08-16T20:11:22Z, ids 33271580906 and 31969817744. Both were dispatched with
event: workflow_dispatch, and WATCHED_DASHBOARD_WORKFLOWS registers that workflow with
event: "schedule" (workflow-watchdog.mjs:5-8), which the run query and the filter at :36-42 apply
before any status test. So those two are outside the watchdog's view for a different reason, and adding
waiting to BLOCKING_RUN_STATUSES would not clear them.

How this relates to existing work

#364 added workflow-watchdog.mjs, merged 2026-09-11T18:18:26Z, and #365 changed the drain's batching,
merged 17 seconds later. This window opened the following evening. #172 anticipates a dispatch being dropped
or coalesced in the webhook function, which is a different path: here the dispatches arrived.

What I cannot tell from outside

• Why 34717262220 stayed at waiting. The job declares environment: protected, so a pending approval is
  one candidate, and I cannot see whether one existed.
• Whether leaving waiting alone is deliberate, for example because cancelling a concurrency-held run has
  consequences I cannot see from here.
• What the 25 displaced dispatches would have processed, so I cannot say what went unprocessed.

[trask]

Thanks for reporting this!

I've manually cancelled the two stuck jobs and sent PR to hopefully avoid in the future #373