Skip to content

fix: dedupe files_touched at the checkpoint write boundary - #1969

Merged
Soph merged 4 commits into
entireio:mainfrom
MuskanPaliwal:fix-checkpoint-dedupe-boundary
Aug 12, 2026
Merged

fix: dedupe files_touched at the checkpoint write boundary#1969
Soph merged 4 commits into
entireio:mainfrom
MuskanPaliwal:fix-checkpoint-dedupe-boundary

Conversation

@MuskanPaliwal

Copy link
Copy Markdown
Contributor

this is the dedupe half of #1937, split into its own pr after @Soph asked to keep the gitrepo fix separate (comment). the underlying report is #1927: a single checkpoint metadata.json grew to 111.79 MB, 94% of it files_touched, holding 1,292,257 entries of which only 67,340 were unique — a 19x duplication factor. once that blob crossed github's 100 MB limit, entire/checkpoints/v1 became permanently unpushable, and the failure only surfaced as a warning tacked onto unrelated pushes.

what this does

writeSessionToSubdirectory used to write opts.FilesTouched into the session metadata exactly as handed to it. it now runs the list through NormalizeFilesTouched first: dedupe, sort, normalize to forward slashes. on the issue's measured numbers that alone takes the blob from 111.79 MB to about 5.5 MB and back under the limit.

a few details that came out of review on #1937 (thanks @ecgang):

  • the helper preserves nil-versus-empty. files_touched is marshaled without omitempty, so nil stays JSON null and a non-nil empty list stays [] — the wire format is unchanged for empty inputs.
  • both live backends inherit the fix through the shared treeWriter (gitRefsStore embeds it), and the test now runs against both the git-branch and git-refs stores instead of leaning on the embedding argument.
  • the fsstore reference backend applies the same normalization, since it's the worked example the next real backend would copy.

why it warns instead of deduping silently

soph's worry on #1937 was that a silent dedupe hides the actual bug, and it's a fair worry: every merge path in the current code already dedupes — i audited them, and @ecgang audited them independently and came up equally empty (their comment on #1927 lists the paths) — so whatever produced the 19x duplication is either a version older than 0.8.42 or a path nobody has found yet. so when the boundary strips duplicates, it now logs a warning with the session id and the reported-versus-unique counts. if a live producer exists, it shows up in .entire/logs instead of disappearing into the dedupe.

tests

  • TestWriteCommitted_DeduplicatesFilesTouched: duplicated input written, permanent record read back deduplicated and sorted, against both git backends. fails without the boundary dedupe.
  • empty-input cases pin the nil-versus-[] behavior.
  • mise run check green: fmt, lint, unit, integration, and the e2e canary.

what this does not do

it can't repair a branch that already carries an oversized blob — git history is immutable, so an affected entire/checkpoints/v1 needs a one-time rewrite, as discussed on the issue. and it deliberately doesn't touch the nested-checkout walking, which is the other (unbounded) half of the issue and lives in #1937.

🤖 Generated with Claude Code

Review feedback: normalize files_touched through an exported helper that
preserves nil-versus-empty so the wire format is unchanged for empty
lists; run the dedup test against the git-refs store as well as the
git-branch store; apply the same normalization in the fsstore reference
backend.

Entire-Checkpoint: 01KZV9MVF38X7SKS7XTDXXBT8Y
Entire-Checkpoint: 01KZVA816GR7Z6GS2N9TA9T2G3
@MuskanPaliwal
MuskanPaliwal requested a review from a team as a code owner August 12, 2026 16:32
@Soph
Soph merged commit dd55d17 into entireio:main Aug 12, 2026
12 checks passed
@Soph

Soph commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants