Skip to content

[OSSIE][SOLID] Add bidirectional Solid semantic model converter - #352

Open
zackm-solid wants to merge 1 commit into
apache:mainfrom
zackm-solid:converter-solid
Open

[OSSIE][SOLID] Add bidirectional Solid semantic model converter#352
zackm-solid wants to merge 1 commit into
apache:mainfrom
zackm-solid:converter-solid

Conversation

@zackm-solid

Copy link
Copy Markdown

Summary

Adds converters/solid, a bidirectional offline converter between an Apache Ossie
semantic model and a Solid semantic model YAML export, following the hub-and-spoke
pattern in converters/README.md, and registers SOLID in that file's vendor table.

No specification change is required: the schema's Vendor type is an open string
("Any string value is accepted"), with the vendor list held as examples rather than an
enum, so core-spec/ is untouched. CI is registered via
.github/workflows/converter-solid-ci.yml, following the existing per-converter
workflows.

Import (Solid → Ossie) preserves every Solid-only construct — example queries, benchmark
questions, quality rank, indexes, sample values, and each column's raw warehouse type —
in custom_extensions[SOLID], so the round trip is lossless. Export (Ossie → Solid)
emits the key order Solid's own exporter uses, and warns rather than silently dropping
the Ossie constructs Solid's format cannot hold.

Design decisions

Three areas required judgement calls. Each is documented at length in the converter
README, and I'd welcome review on all three.

Dialect resolution. Solid's export does not record its source warehouse, but Ossie
requires a dialect on every expression. The converter infers one from the raw column type
vocabulary (NUMBER/TEXT → Snowflake, LONG/MAP → Databricks, INT64/FLOAT64
BigQuery), overridable with --dialect and falling back to ANSI_SQL with a warning
when the vocabulary is ambiguous or two warehouses tie. Type names shared across
warehouses are deliberately excluded from the vote, since they carry no signal.

Metric expressions. Solid stores formulas against bare column names with the owning
table recorded separately; Ossie expects each column qualified with its dataset. The
rewrite is a surgical splice at tokenizer offsets rather than a re-render, because
round-tripping a parsed tree through sqlglot's generator canonicalizes SQL the converter
was only asked to qualify (CAST(x AS FLOAT) returns as CAST(x AS DOUBLE)). The parser
still runs as a cross-check on the token scan, and any disagreement leaves the expression
exactly as written — an unqualified metric is a far smaller problem than a corrupted one.

Cardinality. Ossie encodes it by direction; a Solid relationship is an undirected
pair of column lists. The "one" side is recovered from the declared primary keys, with
the original orientation preserved for export.

Open interop questions

Three cases are pinned as open decisions in tests/test_cross_vendor.py rather than
resolved, because each is a judgement about what an importer owes a foreign model rather
than a defect in the transform. They are the part of this PR most worth discussing:

  • A column with no datatype gets an empty Solid type. datatype is optional in
    the spec and most converters omit it, making this the largest single gap when importing
    a foreign model. It cannot be closed offline — Solid types its columns from the
    warehouse catalog, and the honest fix is to reconcile against the live catalog at
    import time rather than guess in the converter.
  • A field renamed relative to its column loses the underlying column. Given
    name: ticket_number over expression: ss_ticket_number, the alias survives and the
    real column does not. Emitting the underlying name and keeping the alias as a synonym
    would be truer, but it changes which identifier downstream consumers see.
  • A metric written against bare column names reaches Solid with tables: []. Only a
    dataset-qualified reference identifies its owner; resolving bare names against each
    dataset's declared fields would fix most cases, but it is inference, so it is
    deliberately not done silently.

Two further gaps are format-level and noted in the README's "Future effort" section:
verified queries (Solid's example_queries / benchmark_questions) have no home in the
core spec despite the same construct appearing across the ecosystem, and Solid's export
not recording its source warehouse is what forces dialect inference in the first place.

Tests

240 tests covering both directions against three fixtures — the repository's TPC-DS model
expressed as a Solid export, plus a Databricks and a BigQuery model — a foreign-model
fixture for the cross-vendor cases, and schema validation of every converted document
against core-spec/osi-schema.json.

Related Issues

Checklist

Specification

  • Spec changes are included in core-spec/ and follow the existing structure
  • Spec changes have been discussed on the mailing list or in a linked issue
  • Breaking changes to the spec are clearly called out in the summary

No specification changes. The schema's Vendor type accepts any string, so adding a
vendor needs no core-spec/ edit; SOLID is added to the vendor table in
converters/README.md as documentation.

Ontology

  • Ontology changes in ontology/ are consistent with spec changes
  • New or modified terms are defined and documented

No ontology changes.

Converters

  • Converter logic in converters/ is updated to reflect spec or ontology changes
  • New converters include tests under the converter's test directory

Validation

  • Validation rules in validation/ are updated if the spec changed
  • New validation cases are covered by tests

No validation changes; the spec is unchanged. Converted documents are validated against
core-spec/osi-schema.json in the converter's own tests.

Documentation

  • docs/ is updated to reflect any user-facing changes
  • New features or behaviors are documented with examples where appropriate
  • CONTRIBUTING.md is updated if the contribution process changed

Documentation lives in the converter's own README.md — usage, the three design
decisions, and the known gaps — matching where the other converters document themselves.
Root docs/ is a project landing page rather than per-converter reference, so nothing
there needed changing. The contribution process is unchanged.

Examples

  • examples/ are added or updated for any new spec constructs or converter support

No new spec constructs, so root examples/ is unchanged. The converter's fixtures live
under its own tests/fixtures/, alongside the other converters'. One of them is the
repository's own TPC-DS model expressed as a Solid export, so the shared example is
exercised end to end.

Tests

  • All existing tests pass (pytest / CI green)
  • New functionality is covered by tests

Compliance

  • ASF license headers are present on all new source files
  • No third-party dependencies are added without PMC/IPMC approval

Dependencies are sqlglot and PyYAML — both already used by the dbt and gsf
converters, so nothing new enters the project.

Adds converters/solid, a bidirectional offline converter between an Apache
Ossie semantic model and a Solid semantic model YAML export, following the
hub-and-spoke pattern described in converters/README.md.

Import (Solid -> Ossie) preserves every Solid-only construct -- example
queries, benchmark questions, quality rank, indexes, sample values, and each
column's raw warehouse type -- in custom_extensions[SOLID], so the round trip
is lossless. Export (Ossie -> Solid) emits the key order Solid's own exporter
uses, and warns rather than silently dropping the Ossie constructs Solid's
format cannot hold.

Three areas needed design decisions, documented in the converter README:

* Dialect resolution. Solid's export does not record its source warehouse, but
  Ossie requires a dialect on every expression. The converter infers one from
  the raw column type vocabulary (NUMBER/TEXT means Snowflake, LONG/MAP means
  Databricks, INT64/FLOAT64 means BigQuery), overridable with --dialect and
  falling back to ANSI_SQL with a warning.

* Metric expressions. Solid stores formulas against bare column names with the
  owning table recorded separately; Ossie expects them qualified. The rewrite is
  a surgical splice at tokenizer offsets rather than a re-render, because
  round-tripping a parsed tree through sqlglot's generator canonicalizes SQL the
  converter was only asked to qualify. The parser cross-checks the token scan,
  and any disagreement leaves the expression exactly as written.

* Cardinality. Ossie encodes it by direction; a Solid relationship is an
  undirected pair of column lists. The one side is recovered from the primary
  keys, with the original orientation preserved for export.

Tests cover both directions against three fixtures -- the repository's TPC-DS
model expressed as a Solid export, plus a Databricks and a BigQuery model --
and validate every converted document against core-spec/osi-schema.json.

Reviewed-by: Eden Litvin <edenl@getsolid.ai>
@jbonofre
jbonofre self-requested a review September 1, 2026 17:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant