Drain multiple dashboard queue waves per runner

[trask]

A GitHub Actions runner provisioning failure blocked the pull request dashboard queue for about 30 hours. After the blocker was canceled, catching up 2,144 queued events took another six hours because every 16-item batch required a new runner. Later runner-assignment delays repeatedly paused recovery.

This lets each acquired runner process consecutive 16-item waves during a 40-minute processing window. It finishes and acknowledges the current wave before deciding whether to continue. Before claiming another wave, it reserves at least 10 minutes, or 1.5 times the slowest completed wave when that is longer. If there is not enough time, it leaves the remaining items unclaimed and follows the existing finish path, which dispatches a successor when work remains.

• Preserves incremental acknowledgments, lease heartbeats, retry and dead-letter behavior, and publisher locking.
• Mints a repository-scoped GitHub App token for each wave and removes the private key from child-process environments.
• Restores and saves the classification cache once per runner instead of once per batch.

[opentelemetry-pr-dashboard]

Pull request dashboard status

Merged · refreshed 2026-09-11 18:19 UTC

Status above doesn't look right?

• Anything look wrong? Report it with what you expected; it helps us improve the dashboard.

[Copilot]

🟡 Changes recommended

Two moderate unresolved acknowledgment and request-size issues remain.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Enables a single runner to process multiple dashboard queue waves while preserving retries, acknowledgments, token scoping, and cache reuse.

Changes:

• Adds deadline-aware multi-wave draining and retry exclusions.
• Manages per-wave GitHub App tokens and shared classification cache.
• Extends queue clients/APIs and continuation behavior after dead letters.

File summaries

File	Reviewed changes
.github/workflows/pull-request-dashboard-drain.yml	Runs the wave drainer with deadline and cache management.
.github/scripts/pull-request-dashboard/test_rollout.py	Verifies workflow rollout changes.
.github/scripts/pull-request-dashboard/test_queue_worker_client.py	Tests exclusion forwarding.
.github/scripts/pull-request-dashboard/test_drain_queue.mjs	Tests wave, retry, deadline, and token behavior.
.github/scripts/pull-request-dashboard/test_dashboard_queue.mjs	Tests queue exclusion behavior.
.github/scripts/pull-request-dashboard/test_dashboard_queue_worker.mjs	Tests API exclusion forwarding.
.github/scripts/pull-request-dashboard/queue_worker_client.py	Forwards excluded item keys.
.github/scripts/pull-request-dashboard/process_queue_batch.py	Supports continuation after terminal failures. Nit (1 vote): add coverage for dead-letter continuation through main().
.github/scripts/pull-request-dashboard/netlify/lib/dashboard-queue.mjs	Filters excluded items during claims.
.github/scripts/pull-request-dashboard/netlify/functions/dashboard-queue-worker.mjs	Validates exclusion requests.
.github/scripts/pull-request-dashboard/drain_queue.mjs	Implements wave orchestration, retries, tokens, and acknowledgments. Moderate (2 votes): use a serialized-byte budget for exclusions. Moderate (3 votes): handle acknowledgment failures separately from recorded results. Nit (1 vote): add successful-wave lifecycle coverage.

Review details

Suppressed comments (2)

.github/scripts/pull-request-dashboard/drain_queue.mjs:108

• The new success path that creates a wave token, passes it to the batch, reports limits, revokes it, and interprets the results has no test; the added tests only cover token-creation failure. A regression in the per-wave environment, --continue-after-dead-letters, result parsing, or revocation could therefore pass the suite. Add a successful-wave test that captures the child environment and verifies the returned retry/dead-letter counts and cleanup.

    await runCommand(
      "python3",
      [
        path.join(SCRIPT_DIR, "process_queue_batch.py"),
        "--claims",
        claimsPath,
        "--results",
        resultsPath,
        "--max-repositories",
        "4",
        "--queue-endpoint",
        queueEndpoint,
        "--dispatcher-generation",
        String(generation),
        "--worker-id",
        workerId,
        "--continue-after-dead-letters",
      ],
      {
        ...childEnv,
        GH_TOKEN: installationToken.token,
        PR_DASHBOARD_TOKEN: installationToken.token,
      },

.github/scripts/pull-request-dashboard/process_queue_batch.py:618

• The new drain relies on this flag to keep processing later waves after a terminal result, but the Python tests do not exercise main() with a dead-letter result and verify that the flag returns zero while still acknowledging it. Add a regression test here; otherwise a future change to the flag or return path can silently revert the multi-wave behavior to stopping at the first dead letter.

    return 1 if dead_letters and not args.continue_after_dead_letters else 0

• Files reviewed: 11/11 changed files
• Comments generated: 2
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🟡 Changes recommended

A critical retry/dead-lettering flaw and a lease-expiry risk remain unresolved.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

.github/scripts/pull-request-dashboard/drain_queue.py:214

• After a wave is claimed, the dispatcher and item lease monitor is not started until token_client.mint() completes; the previous wave's monitor has already been closed. Both leases are 15 minutes, while the helper performs network fetch calls without a deadline, so a slow GitHub API can let the newly claimed items and dispatcher lease expire before processing begins. The first heartbeat then fails and the subsequent acknowledgment/finish path cannot reliably recover the wave; keep a monitor alive across token minting and the inter-wave transition, or bound these calls below the lease.

        token = token_client.mint(sorted({claim.repository for claim in claims}))
        print(f"::add-mask::{token}")
        processor_env = child_process_environment()
        processor_env.update({"GH_TOKEN": token, "PR_DASHBOARD_TOKEN": token})
        _summary, results = process_claims(

• Files reviewed: 16/16 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

The critical token-isolation and moderate lease-heartbeat issues remain unresolved.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

.github/scripts/pull-request-dashboard/drain_queue.py:208

• Each wave's claims already hold 15-minute item/dispatcher leases, but this mints the token before process_claims creates its LeaseMonitor; after process_claims returns, the rate-limit/revocation calls and the next claim also run without a monitor. A slow GitHub App request (the helper has no explicit timeout) can therefore expire the lease, making fallback acknowledgments stale and preventing finish from handing off the queue until scheduled recovery. Keep a heartbeat alive across token setup/cleanup and inter-wave claim gaps, or otherwise bound and heartbeat these calls.

.github/scripts/pull-request-dashboard/drain_queue.py:89

• The key scheduling branch that reserves 1.5 times the slowest completed wave is not covered: all multi-wave tests use a clock that always returns 0, so every measured duration is zero and this multiplier can never affect whether another claim is made. Add a controlled-clock test with a non-zero wave duration that verifies the next claim is withheld when the calculated reservation reaches the deadline.

        next_wave_seconds = max(
            minimum_wave_seconds,
            maximum_wave_seconds * wave_duration_multiplier,
        )
        if finished_at + next_wave_seconds >= processing_deadline:

• Files reviewed: 16/16 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

Concurrent repository tasks create redundant full-queue lease monitors, multiplying storage traffic and write contention.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 16/16 changed files
• Comments generated: 1
• Review effort level: Balanced

[Copilot]

🔵 Needs a closer look

The multi-wave queue, token, lease, and acknowledgment changes require final human review.

Review details

• Files reviewed: 16/16 changed files
• Comments generated: 0 new
• Review effort level: Lite