[OSSIE][SOLID] Add bidirectional Solid semantic model converter - #352
Open
zackm-solid wants to merge 1 commit into
Open
[OSSIE][SOLID] Add bidirectional Solid semantic model converter#352zackm-solid wants to merge 1 commit into
zackm-solid wants to merge 1 commit into
Conversation
Adds converters/solid, a bidirectional offline converter between an Apache Ossie semantic model and a Solid semantic model YAML export, following the hub-and-spoke pattern described in converters/README.md. Import (Solid -> Ossie) preserves every Solid-only construct -- example queries, benchmark questions, quality rank, indexes, sample values, and each column's raw warehouse type -- in custom_extensions[SOLID], so the round trip is lossless. Export (Ossie -> Solid) emits the key order Solid's own exporter uses, and warns rather than silently dropping the Ossie constructs Solid's format cannot hold. Three areas needed design decisions, documented in the converter README: * Dialect resolution. Solid's export does not record its source warehouse, but Ossie requires a dialect on every expression. The converter infers one from the raw column type vocabulary (NUMBER/TEXT means Snowflake, LONG/MAP means Databricks, INT64/FLOAT64 means BigQuery), overridable with --dialect and falling back to ANSI_SQL with a warning. * Metric expressions. Solid stores formulas against bare column names with the owning table recorded separately; Ossie expects them qualified. The rewrite is a surgical splice at tokenizer offsets rather than a re-render, because round-tripping a parsed tree through sqlglot's generator canonicalizes SQL the converter was only asked to qualify. The parser cross-checks the token scan, and any disagreement leaves the expression exactly as written. * Cardinality. Ossie encodes it by direction; a Solid relationship is an undirected pair of column lists. The one side is recovered from the primary keys, with the original orientation preserved for export. Tests cover both directions against three fixtures -- the repository's TPC-DS model expressed as a Solid export, plus a Databricks and a BigQuery model -- and validate every converted document against core-spec/osi-schema.json. Reviewed-by: Eden Litvin <edenl@getsolid.ai>
jbonofre
self-requested a review
September 1, 2026 17:52
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
converters/solid, a bidirectional offline converter between an Apache Ossiesemantic model and a Solid semantic model YAML export, following the hub-and-spoke
pattern in
converters/README.md, and registersSOLIDin that file's vendor table.No specification change is required: the schema's
Vendortype is an open string("Any string value is accepted"), with the vendor list held as
examplesrather than anenum, socore-spec/is untouched. CI is registered via.github/workflows/converter-solid-ci.yml, following the existing per-converterworkflows.
Import (Solid → Ossie) preserves every Solid-only construct — example queries, benchmark
questions, quality rank, indexes, sample values, and each column's raw warehouse type —
in
custom_extensions[SOLID], so the round trip is lossless. Export (Ossie → Solid)emits the key order Solid's own exporter uses, and warns rather than silently dropping
the Ossie constructs Solid's format cannot hold.
Design decisions
Three areas required judgement calls. Each is documented at length in the converter
README, and I'd welcome review on all three.
Dialect resolution. Solid's export does not record its source warehouse, but Ossie
requires a dialect on every expression. The converter infers one from the raw column type
vocabulary (
NUMBER/TEXT→ Snowflake,LONG/MAP→ Databricks,INT64/FLOAT64→BigQuery), overridable with
--dialectand falling back toANSI_SQLwith a warningwhen the vocabulary is ambiguous or two warehouses tie. Type names shared across
warehouses are deliberately excluded from the vote, since they carry no signal.
Metric expressions. Solid stores formulas against bare column names with the owning
table recorded separately; Ossie expects each column qualified with its dataset. The
rewrite is a surgical splice at tokenizer offsets rather than a re-render, because
round-tripping a parsed tree through sqlglot's generator canonicalizes SQL the converter
was only asked to qualify (
CAST(x AS FLOAT)returns asCAST(x AS DOUBLE)). The parserstill runs as a cross-check on the token scan, and any disagreement leaves the expression
exactly as written — an unqualified metric is a far smaller problem than a corrupted one.
Cardinality. Ossie encodes it by direction; a Solid relationship is an undirected
pair of column lists. The "one" side is recovered from the declared primary keys, with
the original orientation preserved for export.
Open interop questions
Three cases are pinned as open decisions in
tests/test_cross_vendor.pyrather thanresolved, because each is a judgement about what an importer owes a foreign model rather
than a defect in the transform. They are the part of this PR most worth discussing:
datatypegets an empty Solidtype.datatypeis optional inthe spec and most converters omit it, making this the largest single gap when importing
a foreign model. It cannot be closed offline — Solid types its columns from the
warehouse catalog, and the honest fix is to reconcile against the live catalog at
import time rather than guess in the converter.
name: ticket_numberoverexpression: ss_ticket_number, the alias survives and thereal column does not. Emitting the underlying name and keeping the alias as a synonym
would be truer, but it changes which identifier downstream consumers see.
tables: []. Only adataset-qualified reference identifies its owner; resolving bare names against each
dataset's declared fields would fix most cases, but it is inference, so it is
deliberately not done silently.
Two further gaps are format-level and noted in the README's "Future effort" section:
verified queries (Solid's
example_queries/benchmark_questions) have no home in thecore spec despite the same construct appearing across the ecosystem, and Solid's export
not recording its source warehouse is what forces dialect inference in the first place.
Tests
240 tests covering both directions against three fixtures — the repository's TPC-DS model
expressed as a Solid export, plus a Databricks and a BigQuery model — a foreign-model
fixture for the cross-vendor cases, and schema validation of every converted document
against
core-spec/osi-schema.json.Related Issues
Checklist
Specification
core-spec/and follow the existing structureNo specification changes. The schema's
Vendortype accepts any string, so adding avendor needs no
core-spec/edit;SOLIDis added to the vendor table inconverters/README.mdas documentation.Ontology
ontology/are consistent with spec changesNo ontology changes.
Converters
converters/is updated to reflect spec or ontology changesValidation
validation/are updated if the spec changedNo validation changes; the spec is unchanged. Converted documents are validated against
core-spec/osi-schema.jsonin the converter's own tests.Documentation
docs/is updated to reflect any user-facing changesCONTRIBUTING.mdis updated if the contribution process changedDocumentation lives in the converter's own
README.md— usage, the three designdecisions, and the known gaps — matching where the other converters document themselves.
Root
docs/is a project landing page rather than per-converter reference, so nothingthere needed changing. The contribution process is unchanged.
Examples
examples/are added or updated for any new spec constructs or converter supportNo new spec constructs, so root
examples/is unchanged. The converter's fixtures liveunder its own
tests/fixtures/, alongside the other converters'. One of them is therepository's own TPC-DS model expressed as a Solid export, so the shared example is
exercised end to end.
Tests
pytest/ CI green)Compliance
Dependencies are
sqlglotandPyYAML— both already used by thedbtandgsfconverters, so nothing new enters the project.