refactor(cloud): split cloud object store integration into a vortex-cloud crate

[robert3005]

Rationale for this change

Follow-up to #8845, which added the OpenDAL-backed cos:// store.

URL-to-ObjectStore resolution had grown three independent implementations:

Caller	Schemes	Caching	Credentials
vortex-python's Registry	file, s3, azure, gcs, http, cos	per-prefix	case-insensitive env
vortex-jni's make_object_store	file, s3, azure, gcs, cos	per-URL	property-driven builders
vortex-duckdb's resolve_filesystem	file, s3 only	none	from_env only

Only the first knew about the OpenDAL-backed schemes, and it was pub(crate), so no Rust
consumer could reach it. Adding a provider meant touching every caller — which is exactly what
#8845 had to do, and what the next provider would have to do again.

This also unblocks a concrete gap: vortex-object-store-opendal was publish = false, so no
published crate could depend on it.

What changes are included in this PR?

A new vortex-cloud crate owns all cloud object store integration. The registry and the
OpenDAL-backed services now live together in one place:

• vortex-python's private Registry moves in and becomes public vortex_cloud::Registry
  (309 lines deleted from the binding).
• vortex-object-store-opendal is folded in as a vortex_cloud::opendal module and deleted as a
  standalone crate. Files move with git mv, so history follows.
• vortex-io sheds its registry, its object_store_registry / opendal features, and its url
  dependency. It keeps the three generic object_store adapters
  (ObjectStoreFileSystem, ObjectStoreReadAt, ObjectStoreWrite), which are provider-agnostic —
  they serve LocalFileSystem and InMemory exactly as they serve S3 — and compat/obj_store.rs,
  whose impl ObjectStore for Compat<T> cannot move (orphan rule: Compat is defined in
  vortex-io).
• vortex-cloud depends on nothing in vortex-io, so the split adds no dependency edge to
  vortex-file, vortex-datafusion, or the benchmarks.

The bindings resolve through the facade's new vortex::cloud:

• vortex-duckdb replaces its file/s3-only match with the registry, gaining Azure, GCS, HTTP
  and per-bucket client reuse.
• vortex-jni keeps its property-driven builders — they have S3 behaviour the registry does not
  reproduce — but asks supports_scheme instead of hard-coding "cos", so the call site stays
  correct as services are added.

Alibaba Cloud OSS, alongside COS, behind per-service cos / oss features. OSS reads
ALIBABA_CLOUD_* (with the underscore) and has no disable_config_load; its nearest knob,
skip_signature, is exposed for public buckets.

Feature-gating. The registry sits behind vortex-cloud/registry so plain file IO does not pull
in the cloud HTTP stack: vortex-file with object_store still resolves zero reqwest. Making the
services opt-in (default = []) also made the crate wasm32-clean, which removed a CI exclusion —
see below.

Remove ambient process state from tests. The registry takes its configuration from an injected
EnvSource, replacing three unsafe std::env::set_var blocks. set_var became unsafe in Rust
2024 precisely because cargo test runs tests on multiple threads in one process.

Merge of develop resolves a modify/delete conflict against #8733, which fixed a
percent-encoding bug in the file this PR hoists. The deletion is kept and the fix ported into
vortex-cloud: resolve saturates the segment-count subtraction (a percent-encoded
refs%2Fconvert%2Fparquet is one raw URL segment but decodes to three, so the subtraction
underflowed and panicked), and path_suffix decodes through Path::from_url_path. Verified by
reverting the fix and watching test_resolve_percent_encoded_path panic on subtract-overflow.
datasets.py builds exactly that URL shape, so the fix is on the live HuggingFace path.

What APIs are changed? Are there any user-facing changes?

New public Rust API:

• Crate vortex-cloud, re-exported as vortex::cloud under the object_store_registry feature.
• vortex_cloud::Registry (+ Registry::new), behind registry.
• vortex_cloud::opendal::{supports_scheme, SUPPORTED_SCHEMES, COS_SCHEME, CosConfig, make_cos_store, OSS_SCHEME, OssConfig, make_oss_store, make_opendal_store, make_opendal_store_with_env, OpenDALStoreError}, behind cos / oss / opendal.
• vortex-duckdb gains an opendal feature.

Removed: vortex-io's object_store_registry and opendal features, and the
vortex-object-store-opendal crate (never published, so no released API is affected).

No Python-facing API change: vortex-python swaps a private implementation for the shared one.
DuckDB users gain Azure/GCS/HTTP support that previously errored with Unsupported URL scheme.

One CI change worth a look

--exclude vortex-object-store-opendal was added to the wasm32 build because that crate defaulted
to cos + oss and so pulled opendal and reqwest. vortex-cloud defaults to no services, so the
exclusion is dropped rather than renamed. Verified: cargo tree --workspace --target wasm32-unknown-unknown resolves no opendal at all, and cargo check -p vortex-cloud --target wasm32-unknown-unknown succeeds. Left in place, the stale crate name would have silently matched
nothing and the wasm32 job would have built vortex-cloud unguarded.

Checks

Check	Result
cargo test -p vortex-cloud --all-features	22 passed + 2 doctests
cargo test -p vortex-io --all-features	143 passed
cargo test -p vortex-python --all-features --lib	5 passed
cargo check -p vortex-cloud × {no-default, registry, cos, oss, all}	pass
cargo check -p vortex-io --all-features	pass
cargo check -p vortex-file --all-features	pass
cargo check -p vortex-python --all-features --all-targets	pass
cargo check -p vortex-jni --all-features --all-targets	pass
cargo check -p vortex --features files,tokio,object_store,object_store_registry,opendal	pass
cargo check --target wasm32-unknown-unknown -p vortex-cloud -p vortex-io -p vortex-file	pass
cargo clippy -p vortex-cloud -p vortex-io -p vortex-python -p vortex-jni --all-targets --all-features	clean
cargo clippy -p vortex-cloud --no-default-features	clean
cargo +nightly fmt --all	clean
cargo metadata --locked	Cargo.lock up to date
yamllint --strict -c .yamllint.yaml .github/workflows/ci.yml	clean
uv run --all-packages make -C docs clean html	build succeeded (--fail-on-warning)
uv run --all-packages make -C docs clean doctest	271 tests, 0 failures

Could not run locally: cargo check -p vortex-duckdb — the build script downloads DuckDB source
from github.com/duckdb/duckdb/archive, which returns 403 through this environment's proxy
(codeload.github.com too). The vortex-duckdb change is 2 files; I verified by inspection that
vortex::cloud::Registry is re-exported under the features the manifest requests, that
From<object_store::Error> for VortexError exists for the new ?, and that no import became
unused. CI's DuckDB jobs will be the first real compile of it.

cargo deny was also not run (failed to build locally). The new OSS dependencies
(opendal-service-oss, reqsign-aliyun-oss) are both Apache-2.0, matching the COS equivalents
already accepted in #8845.

🤖 Generated with Claude Code

https://claude.ai/code/session_012TyPy9MbKP2j546m1CaKLk

[codspeed-hq]

Merging this PR will not alter performance

✅ 1841 untouched benchmarks
⏩ 55 skipped benchmarks¹

---

_{Comparing claude/opendal-integration-review-1h8f3a (e278343) with develop (2a54a2c)}

Footnotes

1. 55 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[AdamGS]