[stylessh]

First of two PRs that make deploys safe and observable after the 2026-11-08 D1-full incident. This one adds the migration gate, /api/health, the alert watchdog and runbooks. The deploy workflow changes (bookmark, health gate, auto-rollback) follow in a second PR, because the gate depends on /api/health being live.

Migration gate: .github/workflows/migration-check.yml

Runs on PRs that touch apps/dashboard/drizzle/**, src/db/** or drizzle.config.ts, and re-runs when the PR body is edited. scripts/check-d1-migrations.mjs runs four checks:

1. Changed migrations. Fails on DROP, ALTER … DROP COLUMN, ALTER … RENAME, TRUNCATE, or a DELETE/UPDATE without WHERE in the PR's migration files (diffed against the merge commit's first parent), unless the PR body has migration-destructive: approved. Also fails when a PR edits or deletes an already-applied migration, since production never re-runs those (override: migration-edit: approved).
2. Apply to a fresh local D1. wrangler d1 migrations apply --local into a throwaway Miniflare state.
3. drizzle-kit check.
4. Schema drift. drizzle-kit generate from src/db/schema.ts into a temp dir, applied to an in-memory SQLite. Its tables, columns, indexes and foreign keys are compared with the database from step 2. The journal only covers 0000–0001 and later migrations are hand-written, so a generate dry run against the journal would always report drift; this compares actual results instead. The four legacy cache tables are allowlisted until Drop the legacy cache tables after the production purge #198 drops them, and the check says when the allowlist can go.

Locally against main, the gate passes in about 2 s: 7 migrations applied in 1.4 s. I verified each failure mode with throwaway commits:

Case	Result
#198's 0007_drop_legacy_cache_tables.sql, no marker	❌ lists the 4 DROP TABLE statements
same, with migration-destructive: approved in the body	✅ exit 0; also reports the legacy allowlist is no longer needed
actor column added to github_change in the schema, no migration	❌ github_change: schema has `actor text notnull=0 default=null pk=0`, migrations do not
one line appended to 0005_user_forbidden_org.sql	❌ edit to an applied migration

#198 will need migration-destructive: approved in its body. I'll leave a comment there.

GET /api/health

New route, src/routes/api/health.ts, with the logic in src/lib/health.server.ts. Each dependency check is a discriminated result (ok | failed, with latency and error). The top-level status is derived in one place (overallStatus), and the HTTP code comes from it: 200 or 503.

{"status":"ok","checkedAt":"…","version":{"status":"known","id":"285c121f-…","tag":"","uploadedAt":"…"},
 "checks":{"d1":{"status":"ok","latencyMs":1,"sizeBytes":8192},"kv":{"status":"ok","latencyMs":0}},
 "maintenance":{"status":"never_recorded"}}

That is real output from a local production build (wrangler dev on dist/server/wrangler.json).

• d1: SELECT 1, plus meta.size_after.
• kv: one GET of the maintenance heartbeat key.
• maintenance: recent | stale | never_recorded | unreadable. The cron handler in entry-worker.ts writes a heartbeat to KV after each run ({ranAt, result}, 24 h TTL, 288 writes a day). This is informational and never causes a 503, because a missed cron shouldn't roll back a deploy. The watchdog alerts on it instead.
• version: from a CF_VERSION_METADATA binding, which PR B adds. Until then it reports {"status":"unknown"}.

Each check times out after 5 s. The response contains no secrets, only statuses, latencies, DB size and error messages. Test: one vitest covering the degrade-to-503 path (KV throws → 503, kv.failed with the error, d1 still ok, maintenance: unreadable).

The only edit to an existing file is the scheduled handler in entry-worker.ts, merged with #200's error-logging version, plus a KV stub in #200's entry-worker.test.ts fixture.

Alerts: .github/workflows/production-alerts.yml

No Cloudflare Notifications were created. I tried with the owner's wrangler OAuth token: every /alerting/v3/* endpoint returns 10000 Authentication error, because wrangler's OAuth scopes don't include notifications. Workers Observability queries return Forbidden. Separately, Cloudflare's available notifications list has no type for Worker error rate, cron failure or D1 storage.

So this PR implements the design doc's fallback. Every 15 minutes a watchdog evaluates four checks. Each check is firing | clear | unknown, and is reconciled against open issues labelled production-alert, deduplicated by title:

Check	Source	Fires at
/api/health is failing	https://diff-kit.com/api/health, 3 tries 20 s apart	not 200/ok
D1 database above 8 GB	D1 REST API file_size (works when the Worker is down)	≥ 8 GB
scheduled maintenance cron is not running cleanly	/api/health → maintenance	no clean run in 12 min
Worker error rate above 5%	Workers GraphQL analytics, 15-min window ending 5 min ago; counts 5xx responses and thrown invocations	≥ 5% and ≥ 50 requests

• firing opens an issue, or leaves the open one alone.
• clear comments "Recovered" and closes it.
• unknown opens a "(check unavailable)" issue, so a broken watchdog check is never silent. If the check already has a firing issue open, it stays open; a blip doesn't close it.
• Network errors and timeouts become unknown. A failed GitHub write on one check doesn't skip the others; it fails the run.

It uses GITHUB_TOKEN (issues: write) and the existing Cloudflare secrets, with no new credentials. A local dry run against production (issue writes stubbed out) gave: health firing (route not deployed yet), D1 clear at 7.99 GB of 10 GB (79.9%), cron unknown, error rate clear with 0 errors in 5,360 requests. After merge I'll run it via workflow_dispatch and report what it opens.

Runbooks

• docs/runbooks/deploy-rollback.md: deploy phases, manual rollback, deploying with the gate skipped, migration timeouts, restoring D1 to the pre-deploy bookmark.
• docs/runbooks/d1-full.md: symptoms (inserts fail, updates work), measuring, a throttled purge loop (20k rows, 4 s pause, back off on 7429), dropping tables safely, Time Travel restore.

docs/data-fetching.md → "Monitoring and alerting" now describes what exists.

Review fixes

An independent review found these, now fixed in d1874a8:

• a blip closing a firing issue and opening duplicates;
• sum.errors missing caught 500s;
• network errors crashing the watchdog;
• a stale base.sha blaming main's migrations on the PR;
• RENAME and UPDATE without WHERE not flagged;
• DROP/**/TABLE evading the scan;
• partial, DESC and expression indexes ignored by the drift check.

Checks run

• pnpm lint, pnpm check-types, pnpm test pass (dashboard: 41 files, 345 tests) after merging origin/main (Harden server routes: graceful storage failures, auth/redirect fixes, +150 tests #200).
• actionlint 1.7.12 passes on all workflows.

🤖 Generated with Claude Code

[coderabbitai]

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 19 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 2 included reviews currently available.

Learn how review limits work.

Review configuration:

Run configuration

• Source mode: Auto
• Repository: Unavailable
• Branch: Unavailable
• File: No configuration file from this repository applied
• Commit: Unavailable
• Layers: Repository UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: 31b2afee-2704-4171-ba70-44400296bc62

Commits

Reviewing files that changed from the base of the PR and between ae29997 and 6360b72.

Files selected for processing: 13

• .github/workflows/migration-check.yml
• .github/workflows/production-alerts.yml
• apps/dashboard/src/entry-worker.test.ts
• apps/dashboard/src/entry-worker.ts
• apps/dashboard/src/lib/health.server.test.ts
• apps/dashboard/src/lib/health.server.ts
• apps/dashboard/src/routeTree.gen.ts
• apps/dashboard/src/routes/api/health.ts
• docs/data-fetching.md
• docs/runbooks/d1-full.md
• docs/runbooks/deploy-rollback.md
• scripts/check-d1-migrations.mjs
• scripts/production-alerts.mjs

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

• X
• Mastodon
• Reddit
• LinkedIn

[stylessh]

@coderabbitai review

[coderabbitai]

Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.