Skip to content

Support CREATE VECTOR INDEX - #2437

Open
moshap-firebolt wants to merge 3 commits into
apache:mainfrom
moshap-firebolt:bigquery-create-vector-index
Open

Support CREATE VECTOR INDEX#2437
moshap-firebolt wants to merge 3 commits into
apache:mainfrom
moshap-firebolt:bigquery-create-vector-index

Conversation

@moshap-firebolt

@moshap-firebolt moshap-firebolt commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

CREATE VECTOR INDEX creates an index for approximate nearest-neighbor search over an embedding column. It is not specific to one dialect — BigQuery, Oracle, SQL Server, MariaDB and TiDB all have it (so the Generic dialect should too). VECTOR is not a keyword, so parse_create previously failed with Expected: an object type after CREATE, found: VECTOR.

The statement shares a common core across dialects and then diverges into dialect-specific trailers. It is parsed permissively for every dialect.

Common core

CREATE [OR REPLACE] VECTOR INDEX [IF NOT EXISTS] <name> ON <table>(<column | expr>)

VECTOR is treated as a modifier on CREATE INDEX (like UNIQUE), reusing the existing CreateIndex node rather than a new statement variant. It is routed through parse_create_index, so it inherits the standard index trailers — USING <method>, INCLUDE (...), WITH (...), expression targets and index options.

Per-dialect forms

Dialect Form / trailer Status
BigQuery OPTIONS(index_type=…, distance_type=…, ivf_options='{…}'), STORING(col, …) parsed
SQL Server WITH (METRIC=…, TYPE=…, MAXDOP=…), bracket-quoted names parsed
TiDB expression target ((VEC_COSINE_DISTANCE(col))), trailing USING HNSW parsed
Oracle core, expression target, INCLUDE (...) parsed
Oracle ORGANIZATION …, DISTANCE …, WITH TARGET ACCURACY …, PARAMETERS(...) not yet — follow-up
MariaDB bare DISTANCE = … M = … options not yet — follow-up

The two follow-up rows need dedicated grammar with real keyword-collision hazards — WITH TARGET ACCURACY overloads the same WITH keyword as SQL Server's WITH (...), and MariaDB's bare single-letter M collides with identifiers — so they are left out of this change.

Changes

  • Add vector, or_replace, options and storing fields to CreateIndex. Display renders CREATE [OR REPLACE ]VECTOR INDEX … and the STORING(...) / OPTIONS(...) trailers.
  • Accept a WITH (...) clause on a vector index in every dialect (for SQL Server), not only the dialects that enable it for a plain CREATE INDEX.
  • Plain CREATE INDEX is unchanged (the new fields default to false/empty).

Tests

All are round-trip (verified_stmt) tests.

  • tests/sqlparser_common.rs — the generic core plus the shared trailers (INCLUDE, STORING, WITH, USING, expression targets), across all dialects.
  • tests/sqlparser_bigquery.rsOPTIONS(...) with real index_type / distance_type / JSON keys, and STORING(...).
  • tests/sqlparser_mssql.rs — bracket-quoted names with WITH (...).
  • tests/sqlparser_mysql.rs — TiDB's distance-function target with USING.
  • tests/sqlparser_oracle.rs — the core, an expression target and an INCLUDE list.

Docs: BigQuery · Oracle · SQL Server · MariaDB · TiDB

BigQuery can create a vector index for approximate nearest-neighbor search over
an embedding column:

    CREATE [OR REPLACE] VECTOR INDEX [IF NOT EXISTS] <name>
    ON <table>(<column>)
    OPTIONS(index_type = 'IVF', distance_type = 'COSINE', ...)

`VECTOR` is not a keyword, so `parse_create` previously failed with
`Expected: an object type after CREATE, found: VECTOR`.

### Changes
- Add `vector`, `or_replace` and `options` fields to `CreateIndex`. `VECTOR` is
  a modifier on `CREATE INDEX` (like `EXTERNAL` on `CREATE TABLE`), so it reuses
  the existing node rather than a new statement variant. `Display` renders
  `CREATE [OR REPLACE ]VECTOR INDEX ...` plus a trailing `OPTIONS(...)`.
- Parse the form in `parse_create` via a small `parse_create_vector_index`
  helper; the `OPTIONS(...)` clause reuses `parse_options` / `SqlOption`, so it
  parses and renders the same way as `CREATE TABLE` / `CREATE VIEW` OPTIONS.
- Plain `CREATE INDEX` is unchanged (all three fields default to false/empty).
- New test `parse_bigquery_create_vector_index` in `tests/sqlparser_bigquery.rs`
  verifies the round-trip and covers `OR REPLACE`, `IF NOT EXISTS`, multi-part
  names and the OPTIONS-less form.

Docs: https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#create_vector_index_statement

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

@LucaCappelletti94 LucaCappelletti94 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At this time, this PR edits would suggest that CREATE VECTOR INDEX is a BigQuery-specific syntax, but it is not, as it is supported also by:

  • Oracle SQL
  • Microsoft SQL
  • Therefore also Generic Dialect

Therefore, move the tests to sqlparser common and ensure that all dialects that support the syntax successfully parse it. It may be acceptable to maintainers that dialects that do not support the syntax also parse it given the current preference for permissive parsing. I suggest you also review the possible syntax variations that these other dialects may have on this syntax, so as to cover more of its variants completely.

@moshap-firebolt
moshap-firebolt force-pushed the bigquery-create-vector-index branch from 1e9b099 to 8c5ccd7 Compare August 18, 2026 18:51
@moshap-firebolt moshap-firebolt changed the title BigQuery: support CREATE VECTOR INDEX Support CREATE VECTOR INDEX Aug 18, 2026
@moshap-firebolt
moshap-firebolt force-pushed the bigquery-create-vector-index branch 2 times, most recently from de8c8a1 to 1e9b099 Compare August 18, 2026 19:31
@moshap-firebolt moshap-firebolt changed the title Support CREATE VECTOR INDEX BigQuery: support CREATE VECTOR INDEX Aug 18, 2026
moshap-firebolt and others added 2 commits August 18, 2026 16:46
Address review feedback that `CREATE VECTOR INDEX` is not BigQuery-specific
(Oracle, SQL Server, MariaDB and TiDB also have it, hence Generic too).

- Route the statement through `parse_create_index` instead of a separate
  BigQuery helper, so it inherits the standard index trailers (`USING`,
  `INCLUDE`, `WITH`, expression targets, index options) that cover the
  Oracle / SQL Server / TiDB variants, plus the BigQuery `OPTIONS(...)` clause.
- Align `Display` order (OPTIONS after WITH) with parse order so `INCLUDE` +
  `OPTIONS` combinations round-trip.
- Drop the BigQuery-specific doc-comment framing.
- Move the test to `tests/sqlparser_common.rs` as `parse_create_vector_index`,
  running across all dialects and covering `OR REPLACE`, `IF NOT EXISTS`,
  schema-qualified names, `OPTIONS(...)`, the `INCLUDE` trailer, and an
  expression target.

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Extend the shared `CREATE VECTOR INDEX` parsing to the trailers used by the
other dialects, and add per-dialect tests:

- `STORING(...)` covering-column clause (BigQuery); new `storing` field on
  `CreateIndex`, rendered after `INCLUDE`.
- Accept a `WITH (...)` options clause on a vector index in every dialect
  (SQL Server `WITH (METRIC = ..., TYPE = ..., MAXDOP = ...)`), not only the
  dialects that enable it for a plain `CREATE INDEX`.

Tests:
- `tests/sqlparser_common.rs` — the generic core plus the shared trailers
  (`INCLUDE`, `STORING`, `WITH`, `USING`, expression targets) across all dialects.
- `tests/sqlparser_bigquery.rs` — `OPTIONS(...)` with index_type / distance_type
  / JSON tuning keys, and `STORING(...)`.
- `tests/sqlparser_mssql.rs` — bracket-quoted names with `WITH (...)`.
- `tests/sqlparser_mysql.rs` — TiDB's distance-function target with `USING`.
- `tests/sqlparser_oracle.rs` — the core, an expression target and an `INCLUDE`
  list (Oracle's `ORGANIZATION` / `DISTANCE` / `WITH TARGET ACCURACY` /
  `PARAMETERS` clauses are not yet parsed).

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
@moshap-firebolt
moshap-firebolt force-pushed the bigquery-create-vector-index branch from b6aa124 to dd1de78 Compare August 19, 2026 00:27
@moshap-firebolt moshap-firebolt changed the title BigQuery: support CREATE VECTOR INDEX Support CREATE VECTOR INDEX Aug 19, 2026
@moshap-firebolt

Copy link
Copy Markdown
Contributor Author

@LucaCappelletti94 - done, added tests demonstrating different syntaxes. Note - didn't implement full blown Oracle's one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants