Skip to content

fix(core): preserve configured blob payloads - #232

Merged
beinan merged 1 commit into
mainfrom
codex/issue-222-blob-columns
Aug 9, 2026
Merged

fix(core): preserve configured blob payloads#232
beinan merged 1 commit into
mainfrom
codex/issue-222-blob-columns

Conversation

@dcfocus

@dcfocus dcfocus commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • keep configured context payload columns inline and mark them with neutral schema metadata for lazy default projections
  • recover blob-column configuration on reopen and reject legacy blob-v2 schemas with an actionable error
  • preserve lazy payloads during partial updates and materialize text when lexical retrieval needs it
  • add regressions for binary round trips, WAL merge, reopen, updates, text retrieval, and legacy-schema rejection

Compatibility

Datasets already created with lance-encoding:blob cannot be recovered through MemWAL today: reads return no payload and WAL merge fails. This change fails fast when opening those schemas rather than continuing with silent data loss. New datasets store payloads inline and exclude configured blob columns from default bulk projections.

Testing

  • cargo fmt --all -- --check
  • cargo clippy --workspace --all-targets -- -D warnings
  • cargo test --workspace --all-targets
  • cd python && uv run --extra tests pytest (213 passed, 6 skipped, 3 xfailed)
  • cd python && uv run ruff format --check .
  • cd python && uv run ruff check .
  • cd python && uv run --extra dev pyright

Closes #222

@beinan
beinan merged commit 79c1f08 into main Aug 9, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

blob_columns (blob-v2 offload) silently loses data and permanently breaks WAL merge

2 participants