[stylessh]

The change feed was the main D1 cost: about 8M rows written per day for a 3.5 MB database.

Why. Every webhook inserted a row, and the sweeper deleted it 15 minutes later. D1 counts each index touched as another row written, so the insert and the delete cost three rows each, six per change. A production sample from 2026-11-10 had 14,769 rows covering only 865 distinct entities: 94% were repeats, mostly CI (check_run / workflow_job fire dozens of times per commit). One index, (repo, id), was never read by any query.

What changes

• Migration 0008_github_change_latest recreates github_change keyed by (repo, entity_key), with an index on version only.
• A webhook upserts its row. A repeat change updates version and changed_at, which costs about 2 rows written instead of 6.
• Each write takes its version from MAX(version) + position inside its own statement. D1 runs one statement at a time, so a later write always gets a version above anything a reader has seen. No clocks are involved.
• The sweeper keeps the same 15-minute window but never deletes the newest row, so version can't go backwards when traffic is quiet.
• A cursor ahead of the table reads as expired: true. That covers cursors issued before this migration: after the deploy, each open tab refetches its active queries once.

Expected effect. About 6 → 2 rows written per change, so roughly 240M → 75M rows written per month at today's webhook rate. Steady-state table size drops from ~16k rows to ~1k. I'll confirm with wrangler d1 info quickhub-db (rows_written_24h) a day after the deploy.

Deploy notes. The migration drops the old table, which only holds a 15-minute window of refresh signals. Between the migration and the new code going live, change-feed writes fail and are dropped, and reads answer "nothing changed". Both paths are designed to fail that way.

Tests

• Unit tests on real SQLite: repeated changes keep one row and are reported again after each new change; a cursor ahead of the table expires; the sweeper keeps the newest row.
• The change-feed e2e now sends a second change to the same issue and checks the open list refreshes again.
• vitest run (480 passed), and playwright test --project=isolated --no-deps tests/isolated/change-feed.spec.ts passed locally.

migration-destructive: approved

🤖 Generated with Claude Code

Summary by CodeRabbit

• Bug Fixes
  • The change feed now reflects repeated updates to the same item, helping ensure later changes appear without creating duplicate entries.
  • Improved feed refresh and cleanup behavior helps preserve the latest updates and avoid stale results.
• Documentation
  • Updated data-fetching guidance and maintenance instructions to describe the current change-feed behavior.

[coderabbitai]

Caution

Review failed

The pull request is closed.

Walkthrough

The change feed now stores one versioned row per repository entity instead of append-only rows. Writes update existing entity rows, feed cursors use versions, and cleanup preserves the latest-version row. Tests and documentation cover the revised storage, cursor, and cleanup behavior.

Changes: GitHub change feed

Layer / File(s)	Summary
Entity-keyed storage model apps/dashboard/db/migrations/0008_github_change_latest.sql, apps/dashboard/db/migrations/meta/_journal.json, apps/dashboard/src/db/schema.ts, docs/data-fetching.md	The migration and schema define github_change with (repo, entity_key) as its primary key and an index on version. The documentation describes the revised storage model and updated measurements.
Versioned writes, reads, and cursors apps/dashboard/src/lib/github-change.server.ts, apps/dashboard/src/lib/*change*.server.test.ts, apps/e2e/tests/isolated/change-feed.spec.ts, docs/data-fetching.md	Writes upsert rows by entity and assign versions. Reads and cursors use versions, and an ahead-of-table cursor can expire. Tests cover repeated updates, cursor behavior, and webhook refreshes.
Latest-row cleanup and validation apps/dashboard/src/lib/github-change.server.ts, apps/dashboard/src/lib/github-change.server.test.ts, apps/dashboard/src/lib/scheduled-maintenance.server.test.ts, docs/runbooks/d1-full.md	The sweep deletes expired rows only when their version is below the maximum. Tests check that a latest row remains and later changes can be read. The runbook updates its description of retained rows.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Refactor

Possibly related PRs

• stylessh/diffkit#157: Adds the production workflow that applies remote D1 migrations, including the migration introduced here.

Suggested labels: performance

Merge Risk: Low · up to a96b6

Some open tabs may show stale GitHub data after deployment until they refetch on focus or receive another matching change. Expire legacy cursors before merging if that temporary gap is unacceptable.

- [ ] Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

• X
• Mastodon
• Reddit
• LinkedIn