vortex-data · mprammer · Jul 8, 2026
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -64,6 +64,17 @@ jobs:
             -c .yamllint.yaml \
             .github/
 
+  docs-conformance:
+    name: "Docs conformance"
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    steps:
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd  # v6
+      - name: Check docs facts match their source of truth
+        run: python3 scripts/check-docs-conformance.py
+      - name: Self-test the conformance checker (catch matcher regressions)
+        run: python3 scripts/check-docs-conformance.py --self-test
+
   python-lint:
     name: "Python (lint)"
     runs-on: >-

diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml
@@ -20,6 +20,12 @@ jobs:
     timeout-minutes: 30
     steps:
       - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0  # v7
+      - name: Check docs conformance (fail before publishing stale docs)
+        # Runs FIRST, before the changelog generator writes generated release-note pages into docs/
+        # (their historical text would otherwise be scanned and could fail the current-docs lint).
+        run: |
+          python3 scripts/check-docs-conformance.py
+          python3 scripts/check-docs-conformance.py --self-test
       - uses: ./.github/actions/setup-rust
         with:
           repo-token: ${{ secrets.GITHUB_TOKEN }}

diff --git a/README.md b/README.md
@@ -6,7 +6,7 @@
 [![CodSpeed Badge](https://img.shields.io/endpoint?url=https://codspeed.io/badge.json)](https://codspeed.io/vortex-data/vortex)
 [![Crates.io](https://img.shields.io/crates/v/vortex.svg)](https://crates.io/crates/vortex)
 [![PyPI - Version](https://img.shields.io/pypi/v/vortex-data)](https://pypi.org/project/vortex-data/)
-[![Maven - Version](https://img.shields.io/maven-central/v/dev.vortex/vortex-spark)](https://central.sonatype.com/artifact/dev.vortex/vortex-spark)
+[![Maven - Version](https://img.shields.io/maven-central/v/dev.vortex/vortex-spark_2.13)](https://central.sonatype.com/artifact/dev.vortex/vortex-spark_2.13)
 [![codecov](https://codecov.io/github/vortex-data/vortex/graph/badge.svg)](https://codecov.io/github/vortex-data/vortex)
 [![Cite](https://img.shields.io/badge/cite-CITATION.cff-blue)](CITATION.cff)
 

diff --git a/docs/_extra/_redirects b/docs/_extra/_redirects
@@ -0,0 +1,16 @@
+# Cloudflare Pages redirects for docs.vortex.dev.
+# Copied verbatim to the site root via `html_extra_path` in conf.py.
+# More specific rules must come before broader splat rules.
+
+# The internals/serialization page was split across several specification pages
+# (array-format, ipc-format, reading-a-file); send it to the Specification index, which
+# fans out to all of them.
+/developer-guide/internals/serialization      /specification/  301
+/developer-guide/internals/serialization.html /specification/  301
+
+# The specs/ section was renamed to specification/. Cover the section root explicitly
+# (the splat does not match the bare /specs path) before the child-path splat rule.
+/specs            /specification/  301
+/specs/           /specification/  301
+/specs/index.html /specification/  301
+/specs/*          /specification/:splat  301
diff --git a/docs/api/c/index.rst b/docs/api/c/index.rst
@@ -1,9 +1,10 @@
 C API
 =====
 
-The Vortex C API provides a low-level FFI interface to the Vortex library. It is the foundation for
-other language bindings (including C++) and is suitable for embedding Vortex into C applications or
-building higher-level wrappers.
+The Vortex C API provides a low-level FFI interface to the Vortex library. It is the intended
+foundation for other language bindings and is suitable for embedding Vortex into C applications or
+building higher-level wrappers. (The C++ binding currently uses a direct ``cxx`` Rust bridge; a future
+migration to build it on top of this C FFI is planned.)
 
 .. warning::
     This API should be considered entirely unstable. It *will* change. Please reach out if a stable

diff --git a/docs/api/cpp/index.rst b/docs/api/cpp/index.rst
@@ -1,9 +1,10 @@
 C++ API
 =======
 
-The Vortex C++ API provides an idiomatic C++ wrapper around the Vortex C FFI, built using
-`cxx <https://cxx.rs/>`_. It currently supports reading and writing Vortex files and integrates
-with the Arrow C Data Interface via `nanoarrow <https://arrow.apache.org/nanoarrow/>`_.
+The Vortex C++ API is an idiomatic C++ binding built with `cxx <https://cxx.rs/>`_, which generates
+a direct bridge between C++ and Rust. It supports reading and writing Vortex files and integrates
+with the Arrow C Data Interface via `nanoarrow <https://arrow.apache.org/nanoarrow/>`_. A future
+migration to build the C++ API on top of the C FFI is planned.
 
 In the future we will expand the C++ API to cover Vortex's plugin and extension points. Please
 reach out if you are interested in extending Vortex from C++ so we can prioritize these features.

diff --git a/docs/api/java/index.rst b/docs/api/java/index.rst
@@ -39,21 +39,17 @@ They support any Linux distribution with a GLIBC version >= 2.31. This includes
 * Ubuntu 20.04 or newer
 
 
-Usage Example
--------------
-
-Here's a basic example of using the Vortex Java API to read a Vortex file:
-
-.. code-block:: java
-
-    import dev.vortex.api.File;
-    import dev.vortex.api.Array;
+Core API
+--------
 
-    // Open a Vortex file
-    File vortexFile = File.open("path/to/file.vortex");
+The main entry points of the Vortex Java API are:
 
-    // Read arrays from the file
-    Array array = vortexFile.readArray();
+* ``Session`` — the top-level handle that owns data sources and writers.
+* ``DataSource`` — opens a Vortex file, glob, or table via ``DataSource.open(session, uri)`` and
+  exposes its schema and row count.
+* ``Scan`` / ``Partition`` — a scan over a ``DataSource`` iterates its partitions, yielding
+  Arrow-compatible batches.
+* ``VortexWriter`` — writes arrays to a Vortex file.
 
-    // Work with the array data
-    System.out.println("Array length: " + array.getLength());
+See the `Vortex JNI API <../../_static/vortex-jni/index.html>`_ above for the full,
+method-level reference.
diff --git a/docs/api/python/arrays.rst b/docs/api/python/arrays.rst
@@ -22,8 +22,9 @@ Base Class
 Canonical Encodings
 -------------------
 
-Each :class:`~vortex.DType` has a corresponding canonical encoding. These encodings represent the uncompressed version
-of the array, and are also zero-copy to Apache Arrow.
+Each :class:`~vortex.DType` has a corresponding canonical encoding, with the sole exception of ``Union`` (not yet
+canonicalized). These encodings represent the uncompressed version of the array, and are also zero-copy to
+Apache Arrow. The classes below are the canonical encodings currently exposed in Python.
 
 .. autoclass:: vortex.NullArray
      :members:
@@ -37,19 +38,15 @@ of the array, and are also zero-copy to Apache Arrow.
      :members:
 
 
-.. autoclass:: vortex.VarBinArray
-    :members:
-
-
 .. autoclass:: vortex.VarBinViewArray
     :members:
 
 
-.. autoclass:: vortex.StructArray
+.. autoclass:: vortex.FixedSizeListArray
     :members:
 
 
-.. autoclass:: vortex.ListArray
+.. autoclass:: vortex.StructArray
     :members:
 
 
@@ -72,6 +69,14 @@ Utility Encodings
      :members:
 
 
+.. autoclass:: vortex.VarBinArray
+    :members:
+
+
+.. autoclass:: vortex.ListArray
+    :members:
+
+
 Compressed Encodings
 --------------------
 

diff --git a/docs/api/python/index.rst b/docs/api/python/index.rst
@@ -27,6 +27,7 @@ The Python bindings require Python 3.11 or newer. Pre-built wheels are available
 * x86_64 Linux
 * ARM64 Linux
 * Apple Silicon macOS
+* Intel macOS
 
 They support any Linux distribution with a GLIBC version >= 2.17. This includes
 
@@ -41,14 +42,17 @@ Here's a basic example of using the Vortex Python API to write and read a Vortex
 
 .. code-block:: python
 
+    import pyarrow as pa
     import vortex
 
     # Write a Vortex file from a PyArrow table
-    vortex.io.write_path(my_table, "data.vortex")
+    my_table = pa.table({"id": [1, 2, 3], "name": ["a", "b", "c"]})
+    vortex.io.write(my_table, "data.vortex")
 
     # Read a Vortex file
-    dataset = vortex.dataset("data.vortex")
-    table = dataset.to_arrow()
+    vf = vortex.open("data.vortex")
+    reader = vf.to_arrow()  # a pyarrow.RecordBatchReader
+    table = reader.read_all()
 
 
 API Reference

diff --git a/docs/concepts/arrays.md b/docs/concepts/arrays.md
@@ -41,19 +41,22 @@ define their own.
 
 ### Canonical Arrays
 
-In order to avoid having to implement logic for an exponential combination of encodings, Vortex defines one canonical
-encoding per logical data type. All arrays can eventually be decompressed one of these canonical encodings.
+In order to avoid having to implement logic for an exponential combination of encodings, Vortex defines a canonical
+encoding for each logical data type — every type except `Union`, which is not yet canonicalized. Arrays of those types
+can eventually be decompressed to one of the canonical encodings listed below.
 
 | Data Type              | Canonical Encoding   |
 |------------------------|----------------------|
 | `DType::Null`          | `NullArray`          |
 | `DType::Bool`          | `BoolArray`          |
 | `DType::Primitive`     | `PrimitiveArray`     |
-| `DType::UTF8`          | `VarBinViewArray`    |
+| `DType::Decimal`       | `DecimalArray`       |
+| `DType::Utf8`          | `VarBinViewArray`    |
 | `DType::Binary`        | `VarBinViewArray`    |
 | `DType::Struct`        | `StructArray`        |
 | `DType::List`          | `ListViewArray`      |
 | `DType::FixedSizeList` | `FixedSizeListArray` |
+| `DType::Variant`       | `VariantArray`       |
 | `DType::Extension`     | `ExtensionArray`     | 
 
 ### Builtin Arrays
@@ -100,15 +103,15 @@ These can be found in the `encodings/` directory of the Vortex repository.
 Arrays carry their own statistics with them, allowing many compute functions to short-circuit or optimize their
 implementations. Currently, the available statistics are:
 
+* `is_constant`: Whether the array holds only a single unique value (treating nulls as equal).
+* `is_sorted`: Whether the array's values are in ascending order.
+* `is_strict_sorted`: Whether the array's values are in strictly ascending order, with no duplicates.
+* `max`: The maximum value in the array (ignoring nulls).
+* `min`: The minimum value in the array (ignoring nulls).
+* `sum`: The sum of the non-null values in the array.
 * `null_count`: The number of null values in the array.
-* `true_count`: The number of `true` values in a boolean array.
-* `run_count`: The number of consecutive runs in an array.
-* `is_constant`: Whether the array only holds a single unique value
-* `is_sorted`: Whether the array values are sorted.
-* `is_strict_sorted`: Whether the array values are sorted and unique.
-* `min`: The minimum value in the array.
-* `max`: The maximum value in the array.
-* `uncompressed_size`: The size of the array in memory before any compression.
+* `uncompressed_size_in_bytes`: The size of the array in memory before any compression.
+* `nan_count`: The number of NaN values in the array.
 
 ## Execution
 

diff --git a/docs/concepts/dtypes.md b/docs/concepts/dtypes.md
@@ -19,7 +19,7 @@ with many columns.
 
 ## Logical Types
 
-The following table lists the built-in dtypes in Vortex, each of which can be marked as either nullable or non-nullable.
+The following table lists the built-in dtypes in Vortex, each of which can be marked as either nullable or non-nullable (the `Extension` type is the exception — it inherits nullability from its storage type).
 
 | Name            | Domain                                      |
 |-----------------|---------------------------------------------|
@@ -32,11 +32,14 @@ The following table lists the built-in dtypes in Vortex, each of which can be ma
 | `List`          | See [List](#list)                           |
 | `FixedSizeList` | See [List](#list)                           |
 | `Struct`        | See [Struct](#struct)                       |
+| `Union`         | Tagged union of variant types (partial)     |
+| `Variant`       | Semi-structured, JSON-like values (partial) |
 | `Extension`     | See [Extension](#extension)                 |
 
 :::{note}
-There are additional logical types that Vortex does not yet support, for example fixed-length binary, maps, and variants.
-These may be added in future versions.
+Vortex's logical type system is still evolving. Some types (e.g. fixed-length binary and maps) are
+not yet supported, and others (notably `Variant` for semi-structured values, and `Union`) are only
+partially implemented — see the [roadmap](../project/roadmap.md).
 :::
 
 ### Primitive

diff --git a/docs/concepts/file-format.md b/docs/concepts/file-format.md
@@ -7,23 +7,24 @@ and serializes the layout and its segments into a single file.
 The bulk of the file format specification describes the representation of the footer bytes such that the
 layout tree can be reconstructed for scans.
 
-See the [Vortex File Format Specification](../specs/file-format.md) for full details.
+See the [Vortex File Format Specification](../specification/file-format.md) for full details, or
+[Reading a File](../specification/reading-a-file.md) for a step-by-step walkthrough of the read path.
 
 ## Layout Strategy
 
 The default layout strategy for Vortex files is roughly:
 
 * Struct Layout at the top-level to partition by columns
   * Zoned Layout to store pruning statistics for every 8k rows
-      * Chunked Layout to partition the column into 2MB of uncompressed data
+      * Chunked Layout to partition the column into chunks
         * Compressor Layout to apply a compression strategy
-          * Buffered Layout to localize up to 1MB of compressed chunks per column.
+          * Buffered Layout to localize up to 2 MB of compressed chunks per column.
               * Flat Layout to serialize each individual array chunk
 
 This strategy optimizes for analytical query patterns: column pruning avoids reading unused columns,
 zone statistics enable skipping irrelevant row ranges, and buffered chunks allow efficient I/O
-with parallel decompression. The 8k row zones and 2MB chunks balance pruning granularity against
-metadata overhead.
+with parallel decompression. The 8k row zones and buffered chunk locality balance pruning
+granularity against metadata overhead.
 
 ## Compression Strategies
 

diff --git a/docs/concepts/index.md b/docs/concepts/index.md
@@ -24,7 +24,8 @@ without dictating physical layout, allowing the same logical data to use differe
 
 **[Arrays](arrays.md)** are the in-memory representation. Unlike Arrow, Vortex arrays can be
 *compressed*—an integer array might be bit-packed rather than stored as a flat buffer. Arrays
-share the same representation on disk and over the wire, enabling zero-copy I/O.
+share the same representation in memory, on disk, and over the wire; the on-disk file layout adds
+alignment padding so reads can be zero-copy.
 
 **[Compute](expressions.md)** functions operate directly on compressed arrays where possible,
 dispatching to encoding-specific kernels or falling back to canonical implementations.
@@ -34,10 +35,11 @@ dispatching to encoding-specific kernels or falling back to canonical implementa
 **[Layouts](layouts.md)** organize arrays into larger-than-memory datasets (e.g., chunked row groups)
 and can read from any block storage: local disk, object stores, caches, etc.
 
-**[File Format](../specs/file-format.md)** (`.vortex` files) serialize layouts to disk with efficient
-segment retrieval, FlatBuffer metadata for O(1) schema access, and support for memory mapping.
+**[File Format](../specification/file-format.md)** (`.vortex` files) serialize layouts to disk with
+efficient segment retrieval, FlatBuffer metadata for O(1) schema access, and support for memory
+mapping.
 
-**[IPC Format](../specs/ipc-format.md)** provides streaming transfer of compressed arrays.
+**[IPC Format](../specification/ipc-format.md)** provides streaming transfer of compressed arrays.
 
 ## Integrations
 

diff --git a/docs/concepts/scanning.md b/docs/concepts/scanning.md
@@ -1,8 +1,8 @@
 # Scan API
 
 :::{note}
-The Scan API is on the roadmap and under active development. The core `Source` trait and scan pipeline
-are functional, but the full API surface is not yet fully defined or implemented.
+The Scan API is on the roadmap and under active development. The core `DataSource` trait and scan
+pipeline are functional, but the full API surface is not yet fully defined or implemented.
 :::
 
 The Vortex Scan API defines a standard interface between data storage and query engines. It solves the
@@ -18,11 +18,9 @@ interface that both sides can implement against.
     Iceberg Tables ──► └──────────────┘ ──►  Spark
 ```
 
-Storage backends implement the `Source` trait for reads. Query engines issue a scan request
+Storage backends implement the source interface for reads. Query engines issue a scan request
 describing the filter and projection to push down, and the source returns a stream of
-independently-executable splits that can be run concurrently to produce result arrays. An
-equivalent `Sink` trait exists for the write path, accepting an array stream and writing it to
-the underlying storage.
+independently-executable splits that can be run concurrently to produce result arrays.
 
 ## Motivation
 
@@ -38,15 +36,13 @@ any decompression step.
 
 ## Source
 
-A **Source** represents any scannable tabular data. It accepts a scan request (filter, projection,
-limit) and returns a stream of independently-executable splits. An equivalent **Sink** interface
-exists for the write path, allowing query engines to both read from and write to any storage
-backend through a single pair of interfaces.
+A **source** (the `DataSource` trait) represents any scannable tabular data. It accepts a scan request
+(filter, projection, limit) and returns a stream of independently-executable splits.
 
 ### Splits
 
-A source produces splits, each representing an independent unit of work that can be executed in
-parallel. A split typically corresponds to a range of rows in a layout, such as a chunk or a set
+A source produces splits (the `Partition` trait), each representing an independent unit of work that
+can be executed in parallel. A split typically corresponds to a range of rows in a layout, such as a chunk or a set
 of row-group partitions.
 
 Each split carries size and row count estimates that query engines use for scheduling decisions.
@@ -56,7 +52,7 @@ Splits can also be serialized for distributed execution across remote workers.
 
 A source may front remote storage rather than local files. In this case, the split's execution
 issues a remote call and receives the result over the network. The
-[Vortex IPC format](../specs/ipc-format.md) can be used as the wire protocol for these calls, allowing
+[Vortex IPC format](../specification/ipc-format.md) can be used as the wire protocol for these calls, allowing
 compressed arrays to be transferred without decompression. This gives remote sources the same
 zero-decompression benefits as local scans -- the data stays in its compressed encoding end-to-end,
 from remote storage through the network and into the query engine.
@@ -89,5 +85,5 @@ pipeline.
 
 Query engines integrate with the Scan API by translating their internal plan representations into
 scan requests and consuming the resulting array stream in their preferred format. Integrations
-exist for DuckDB, DataFusion, Spark, and Trino, with each engine converting its native filter
-and projection representations into Vortex [expressions](expressions.md).
+exist for DuckDB, DataFusion, and Spark, each engine converting its native filter and projection
+representations into Vortex [expressions](expressions.md).