fix: dedupe files_touched at the checkpoint write boundary - #1969
Merged
Soph merged 4 commits intoAug 12, 2026
Conversation
Review feedback: normalize files_touched through an exported helper that preserves nil-versus-empty so the wire format is unchanged for empty lists; run the dedup test against the git-refs store as well as the git-branch store; apply the same normalization in the fsstore reference backend. Entire-Checkpoint: 01KZV9MVF38X7SKS7XTDXXBT8Y
Entire-Checkpoint: 01KZVA816GR7Z6GS2N9TA9T2G3
Soph
approved these changes
Aug 12, 2026
Collaborator
|
Thanks! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
this is the dedupe half of #1937, split into its own pr after @Soph asked to keep the gitrepo fix separate (comment). the underlying report is #1927: a single checkpoint
metadata.jsongrew to 111.79 MB, 94% of itfiles_touched, holding 1,292,257 entries of which only 67,340 were unique — a 19x duplication factor. once that blob crossed github's 100 MB limit,entire/checkpoints/v1became permanently unpushable, and the failure only surfaced as a warning tacked onto unrelated pushes.what this does
writeSessionToSubdirectoryused to writeopts.FilesTouchedinto the session metadata exactly as handed to it. it now runs the list throughNormalizeFilesTouchedfirst: dedupe, sort, normalize to forward slashes. on the issue's measured numbers that alone takes the blob from 111.79 MB to about 5.5 MB and back under the limit.a few details that came out of review on #1937 (thanks @ecgang):
files_touchedis marshaled withoutomitempty, so nil stays JSONnulland a non-nil empty list stays[]— the wire format is unchanged for empty inputs.treeWriter(gitRefsStoreembeds it), and the test now runs against both the git-branch and git-refs stores instead of leaning on the embedding argument.why it warns instead of deduping silently
soph's worry on #1937 was that a silent dedupe hides the actual bug, and it's a fair worry: every merge path in the current code already dedupes — i audited them, and @ecgang audited them independently and came up equally empty (their comment on #1927 lists the paths) — so whatever produced the 19x duplication is either a version older than 0.8.42 or a path nobody has found yet. so when the boundary strips duplicates, it now logs a warning with the session id and the reported-versus-unique counts. if a live producer exists, it shows up in
.entire/logsinstead of disappearing into the dedupe.tests
TestWriteCommitted_DeduplicatesFilesTouched: duplicated input written, permanent record read back deduplicated and sorted, against both git backends. fails without the boundary dedupe.[]behavior.mise run checkgreen: fmt, lint, unit, integration, and the e2e canary.what this does not do
it can't repair a branch that already carries an oversized blob — git history is immutable, so an affected
entire/checkpoints/v1needs a one-time rewrite, as discussed on the issue. and it deliberately doesn't touch the nested-checkout walking, which is the other (unbounded) half of the issue and lives in #1937.🤖 Generated with Claude Code