feat(engine,daemon): content-addressed blob store with attachment records and GC

[Zerlight]

Summary

Phase 3 of CODE-627 — Conversation turn graph & immutable attachment store. Linear: https://linear.app/arcbox/issue/CODE-635/featenginedaemon-content-addressed-blob-store-with-attachment-records

Stack: #512 ← this PR (ruocheng/code-635, base ruocheng/code-634) ← #514. Merge bottom-up; this PR's diff is only its own commits.

Phase 3 storage: a per-profile filesystem CAS (blobs/sha256/…, streamed into blobs/tmp/ and published by one atomic rename), attachment records and upload leases on the graph-store connection, reference-rooted GC (committed refs ∪ unexpired leases ∪ open operations, grace period plus boot mark-and-sweep, unlinks serialized with publish), and session resource bytes migrated onto the store.

Commits

• feat(schema,engine): attachment records and a filesystem blob store
• feat(engine): attachment store, upload leases, and reference-rooted GC
• feat(daemon): attachment tables, upload leases, and lease claims on the graph connection
• feat(engine): store session resource bytes in the blob store
• fix(engine): serialize blob GC unlinks with publish and persist resources after the blob row

Verification

Every commit passed pnpm check:ci and pnpm test at its own tip; the stack tip (1d942a62; the same tree as the originally gated 73fc5ff2 plus the two review fixes below on 628/629) is at pnpm check:ci 0 errors, pnpm test 3408 passed / 1 skipped. Adversarial reviewers (one per axis, isolated read-only worktrees) reviewed the branch; each P1/P2 was reproduced with a failing test or a probe step before its fix — the round-by-round record is in the Linear issue's comments.

Checklist

• pnpm check:ci and pnpm test both pass (no Rust changes)
• I ran the affected surface and observed the change working — engine exercised through its harness tests and the dev-mock parity suite; the live-daemon run against real providers is recorded per issue in Linear (the claude edit path still needs a paid turn, opencode its binary)
• Wire: no wire change
• New code and assets are my own work
• Docs and comments are updated where behavior changed (AGENTS.md and module docs in this branch)

[linear-code]

CODE-635

[pullfrog]

Your Claude subscription has hit its usage limit. It resets at 11:30am (UTC). Re-trigger Pullfrog after the reset, or add an ANTHROPIC_API_KEY repo secret — Pullfrog routes around an exhausted subscription automatically when one is present.

Add repo secret → · Model settings → · Setup docs → · Ask in Discord →

  ^{｜ Rerun failed job ➔ ｜ View workflow run ｜ via Pullfrog ｜ Using Claude Opus ｜ 𝕏}

[greptile-apps]

Greptile Summary

This PR introduces a per-profile content-addressed attachment store, persistent attachment and upload-lease records, reference-rooted garbage collection, and migrates uploaded session-resource bytes into the blob store.

• Adds SHA-256-addressed filesystem staging, integrity verification, deduplication, and atomic publication.
• Adds SQLite attachment, blob, variant, lease, and session-resource reference persistence.
• Adds boot and periodic garbage collection rooted by prompts, resources, and unexpired leases.
• Integrates CAS-backed resources into engine startup, upload, hosting, removal, and session lifecycle.
• One failure-cleanup path can delete an existing shared blob after a deduplicated upload.

Confidence Score: 4/5

The PR is not safe to merge until failed deduplicated uploads stop deleting blob files that remain referenced by existing attachments.

The CAS correctly reuses identical existing files, but the upload catch path cannot distinguish a newly published blob from a deduplication hit and unconditionally removes the shared destination after attachment metadata persistence fails.

Files Needing Attention: packages/host/engine/src/resource/service.ts

Important Files Changed

Filename	Overview
packages/host/engine/src/resource/service.ts	Migrates uploaded resources to CAS-backed attachments, but failure cleanup can delete a deduplicated blob still used by existing resources.
packages/host/engine/src/attachment/blob-store.ts	Implements staged SHA-256 verification, immutable publication, deduplication, enumeration, and deletion.
packages/host/engine/src/attachment/gc.ts	Adds serialized boot and periodic mark-and-sweep collection with a grace period.
apps/daemon/src/attachment-store.ts	Persists attachment graphs and leases and transactionally identifies unrooted metadata for collection.
apps/daemon/src/conversation-store.ts	Persists prompt attachment roots and atomically releases claimed upload leases.
apps/daemon/drizzle/0014_elite_mister_fear.sql	Adds attachment, blob, variant, and upload-lease tables plus session-resource attachment references.
packages/foundation/schema/src/model/attachment.ts	Defines validated blob IDs, attachment records, metadata limits, and upload leases.
packages/host/engine/src/engine.ts	Wires attachment storage, a shared I/O mutex, boot sweeping, and periodic GC into engine lifecycle.

Sequence Diagram

sequenceDiagram
  participant Client
  participant Resource as ResourceService
  participant CAS as FsBlobStore
  participant DB as AttachmentStore
  participant GC as AttachmentGc

  Client->>Resource: Upload resource bytes
  Resource->>CAS: Stage and verify SHA-256
  CAS->>CAS: Atomically publish or reuse existing blob
  Resource->>DB: Commit blob and attachment records
  alt metadata commit succeeds
    Resource->>DB: Save session-resource root
    Resource-->>Client: resource.uploaded
  else metadata commit fails
    Resource->>CAS: Delete hash-derived blob
    Note over CAS: Currently also deletes a reused blob<br/>referenced by earlier attachments
  end
  GC->>DB: Delete unrooted metadata after grace period
  GC->>CAS: Unlink blobs with no remaining row

Loading

_{Reviews (1): Last reviewed commit: "fix(conversation): restore persist and r..." | Re-trigger Greptile}

[greptile-apps]

Greptile has paused reviews on this repository — it used its 100 free open-source review credits for this billing period. Reviews resume automatically on September 23. To continue before then, an organization admin can keep reviews running past the free credits — those bill as normal usage.

[pullfrog]

Your Claude subscription has hit its usage limit. It resets at 11:30am (UTC). Re-trigger Pullfrog after the reset, or add an ANTHROPIC_API_KEY repo secret — Pullfrog routes around an exhausted subscription automatically when one is present.

Add repo secret → · Model settings → · Setup docs → · Ask in Discord →

  ^{｜ Rerun failed job ➔ ｜ View workflow run ｜ via Pullfrog ｜ Using Claude Opus ｜ 𝕏}

[pullfrog]

Your Claude subscription has hit its usage limit. It resets at 11:30am (UTC). Re-trigger Pullfrog after the reset, or add an ANTHROPIC_API_KEY repo secret — Pullfrog routes around an exhausted subscription automatically when one is present.

Add repo secret → · Model settings → · Setup docs → · Ask in Discord →

  ^{｜ Rerun failed job ➔ ｜ View workflow run ｜ via Pullfrog ｜ Using Claude Opus ｜ 𝕏}

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[Zerlight]

Review round on this PR (Greptile, one inline thread): the shared-blob unlink after a failed dedupe upload was real at this head and is already gone above it on the stack — the path moved into AttachmentIngest.store (ca40e01, #516), which unlinks only when no blob row exists. fdda262 (top of the stack) adds the pinning test for the ingest path. Gates at the new tip 8278289: check:ci 0 errors, 3421 tests. Detail on the thread; the PR body is unchanged.