Skip to content

fix: bump Lance for custom S3 endpoints - #233

Open
dcfocus wants to merge 2 commits into
mainfrom
codex/issue-197-s3-custom-endpoints
Open

fix: bump Lance for custom S3 endpoints#233
dcfocus wants to merge 2 commits into
mainfrom
codex/issue-197-s3-custom-endpoints

Conversation

@dcfocus

@dcfocus dcfocus commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • bump Rust Lance crates to 9.0.0 and pylance to 9.x so MemWAL preserves storage/session parameters for derived WAL paths
  • update Lance 9 scanner API call sites and make WAL-backed pagination deterministic with a narrow id sort before wide-row fetches
  • make WAL-to-base cleanup idempotent under racing/interrupted merges by merge-inserting newest rows by key and skipping stale prepared commits
  • remove the S3 custom-endpoint xfails and verify reopened Context handles can read moto-backed S3 writes
  • handle Lance 9 internal MemWAL _tombstone column during WAL-to-base merges

Closes #197.

Upstream fix: lance-format/lance#7735, released in Lance v9.0.0.

Testing

  • uv run --extra tests pytest -q
  • uv run --extra dev ruff format --check .
  • uv run --extra dev ruff check .
  • uv run --extra dev pyright
  • uv lock --check
  • cargo-fmt --all -- --check
  • cargo clippy --workspace --all-targets -- -D warnings
  • cargo clippy -p lance-context-core --all-targets -- -D warnings
  • cargo clippy -p lance-context-master --all-targets -- -D warnings
  • cargo check -p lance-context-core
  • cargo check -p lance-context-master
  • cargo test -p lance-context-master --all-targets
  • cargo test -p lance-context-core --all-targets
  • cargo test -p lance-context-core rollout_store::tests::wal_and_all_pagination_take_only_page_rows
  • cargo test -p lance-context-core rollout_store::tests::close_drains_resident_writer_and_add_reopens
  • cargo test -p lance-context-core --test wal_merge_concurrency concurrent_merges_do_not_duplicate_rows

@dcfocus
dcfocus force-pushed the codex/issue-197-s3-custom-endpoints branch from e050775 to 9fb270b Compare August 4, 2026 23:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Writes fail against any custom-endpoint S3 (MinIO/Ceph/R2/moto): MemWAL writer drops storage_options

1 participant