feat: add lightdash converter - #360
Open
Spissable wants to merge 35 commits into
Open
Conversation
Bidirectional converter between Ossie documents and Lightdash semantic definitions (dbt schema.yml meta blocks): datasets/fields/metrics/ relationships map to models/columns/meta.metrics/meta.joins, and Lightdash presentation attributes round-trip through custom_extensions with vendor_name "lightdash". Follows the apache-ossie-<vendor> uv packaging layout and ships tests off the TPC-DS example, including a structural round-trip.
…elity
- Add converter-lightdash-ci.yml mirroring the other converters (tests
were otherwise not exercised by CI)
- Rewrite ${other_table.column} references into cross-dataset
references on import, not just ${TABLE}.column
- Keep inexpressible metric types (percentile) in the extension even
when the metric carries custom SQL, on both column-level and
model-level metrics
- Drop the redundant per-converter .gitignore (covered by the root
one) and unused helpers; document the not-yet-carried model-level
meta keys in the README
…ip columns - Drop structural keys (sql/label on dimensions, sql/description on metrics) from the extension overlay on export, so a hand-authored extension can never override the OSI-derived definition; metric type stays overridable as the documented channel for inexpressible types - Skip relationships whose from_columns/to_columns lengths differ, reporting RELATIONSHIP_COLUMNS_MISMATCHED instead of silently truncating the join - Document that dataset names are not preserved when they differ from the source table name
… model metrics, add [tool.uv] - Surface unparseable lightdash extension data as an EXTENSION_DATA_INVALID issue instead of silently dropping it - Skip model-level metrics without sql on import, reporting METRIC_SQL_MISSING rather than emitting an empty OSI expression - Add the [tool.uv] block (required-version, default-groups) used by the other converters so uv sync reliably installs the dev group in CI
Mirror the change from the recent CI actions path cleanup: trigger on this workflow's own file instead of the whole .github tree.
* upstream/main: CI for databricks converter (apache#261) fix(dbt): resolve SUM_BOOLEAN semantic model for qualified columns (apache#292) [OSSIE][NVIDIA_GSF] Add bidirectional GSF semantic model converter (apache#247) Fix typo in python README.md (apache#284) Bump Java from 17 to 21 (apache#283) Adding AI disclosures updates (apache#259) Fix Omni pytest (apache#263) Updating CI Actions path (apache#276) Bump java version from 11 to 17 for Polaris converter (apache#278) Make the spec's own examples validate against osi-schema.json (apache#209) [Ontology] Flatten concept declarations in ontology spec (apache#257) feat(cli): add plugin invocation protocol (apache#155) Add datatype field to Field and Metric; reframe is_time as role marker (apache#113) Cleanup impl dir (apache#252)
Spec change apache#113 added `datatype` to fields and metrics and reframed `dimension.is_time` as a role marker rather than a type flag, which broke the TPC-DS round-trip: the converter inferred `is_time` from the Lightdash type and wrote `false` where the source left it unset. - Map Lightdash dimension types to and from `datatype`, so the type is carried in standard vocabulary instead of a vendor extension - Leave `is_time` unset on import (Lightdash has no equivalent marker) and report TIME_ROLE_NOT_REPRESENTABLE when a non-temporal field is flagged as a time axis on export - Emit a Lightdash type only for fields that are dimensions, so a measure-only field is not turned into a dimension by a round-trip - Document that datatypes round-trip by category rather than exact type
…name Upstream apache#288 renamed the Python models (OSIDocument -> OssieDocument, ...) and the osi_to_<vendor> / <vendor>_to_osi module convention. Apply the same rename to the Lightdash converter so it imports and runs against main.
The import command dumped the pydantic model without mode="json", so a .yaml target raised RepresenterError on OssieDialect. Take argv in main() and cover both output formats with a CLI test.
…port
Real Lightdash projects alias most joins, reference sibling fields as
${column}, other metrics as ${metric} and use typed metrics over custom SQL.
The importer matched joins on the model name only, left every non-${TABLE}
reference in the Ossie expression, ignored the sql of typed metrics and
emitted percentile and distinct metrics as a bare column.
- resolve join aliases when parsing sql_on; stash the alias and other join
attributes in the relationship's lightdash extension and restore them on
export, aliasing repeated dataset pairs so Lightdash can compile them
- rewrite ${column}, ${TABLE} and ${alias.column} references; inline
${metric} references; skip elements that use parameters or user
attributes with an EXPRESSION_NOT_PORTABLE issue
- express every typed metric as an aggregation over its operand, including
SUM(DISTINCT ...), AVG(DISTINCT ...) and PERCENTILE_CONT ... WITHIN GROUP,
and parse those shapes back into typed metrics on export
On the 37-model jaffle-shop project this takes relationships from 1 of 21 to
21 of 21 and removes every unrewritten reference from the document.
…oins
- export prefers --dialect, then ANSI_SQL, and reports DIALECT_UNAVAILABLE
instead of silently taking whatever dialect comes first
- import labels expressions with --dialect, since Lightdash SQL is written
for the project's warehouse rather than ANSI
- a metric spanning several datasets is hosted on the dataset that joins
all the others directly, with ${joined_model.column} references, instead
of being dropped; TPC-DS's two cross-dataset metrics now export
- field expressions resolve joined datasets the same way and report
FIELD_REFERENCE_UNJOINED when no join exists
Lightdash has vocabulary for more of the Ossie model than the README claimed: - dataset.primary_key <-> meta.primary_key (string or composite list) - ai_context <-> ai_hint on models, dimensions and metrics; multi-line instructions become a list of hints, structured synonyms and examples are rendered as extra hints on export - exported joins carry relationship: many-to-one, the cardinality an Ossie relationship defines, unless a stashed Lightdash value overrides it - time_intervals: OFF <-> an explicit is_time: false on a temporal column - metric datatype is derived from the aggregation and column type on import A label or ai_context on a measure-only field is now dropped with a FIELD_ATTRIBUTE_NOT_REPRESENTABLE issue instead of turning the field into a dimension on export.
Lightdash evaluates {% ... %} / {{ ... }} tags and bare ld.parameters,
ld.query and ld.user references at query time; a dimension or metric using
them is skipped with EXPRESSION_NOT_PORTABLE like ${ld....} references.
dbt 1.10+ nests meta under config; Lightdash merges meta and config.meta with config.meta winning. The importer only read the top-level key and saw an empty project when every model used the new placement. Export gains --meta-under-config to write that placement.
Lightdash scopes metric names per model, Ossie per semantic model, so any real project (count, total_events, unique_users on many models) produced an invalid document. The Ossie name is now Lightdash's own field id, <model>_<metric>, which is stable regardless of what other models define and matches the id users see in the API and URLs. The bare name travels in the lightdash extension and is restored on export; an Ossie metric without a stash exports under its name minus a <model>_ prefix. Names that still collide after qualification are suffixed with a METRIC_NAME_COLLISION issue.
…rgets Lightdash treats dbt seeds as models, so a schema file's seeds: entries become datasets too. A join whose target is not in the input would leave a dangling relationship (the validator rejects it); it is skipped with a JOIN_TARGET_UNKNOWN issue instead.
Lightdash SQL may name a column of the current model without ${TABLE}
(SUM(budget_use)); the Ossie expression then carried no dataset qualifier
and export could not place the metric in a multi-dataset document (67 of
390 jaffle-shop metrics were dropped). Bare identifiers that name one of
the model's columns are now qualified, outside string literals and never
when called as a function, and the hosting model is stashed in the
extension so an expression that names no dataset at all is still placed.
…ield
Lightdash explores compose joins per base model and carry model meta with
no Ossie vocabulary; both were dropped silently. Now:
- a join through another joined model (${projects.org_id} =
${organizations.org_id} on the queries explore) derives the edge between
the two models it names, unless that model declares it itself, and the
join is stashed verbatim on the dataset's lightdash extension; expression
joins and joins with extra conditions are stashed the same way
- export restores stashed joins, replacing the generated join to the same
target and alias, and lets field and metric expressions resolve through
them
- model meta without Ossie vocabulary (label, hidden, sql_filter,
group_details, default_time_dimension, required_filters, ...) and column
meta outside dimension/metrics (additional_dimensions, ...) travel in the
dataset's and field's extension and come back as they were
- aliased joins on the same columns stay distinct relationships; pair and
side order in sql_on no longer force a stash
On the analytics project every one of 140 joins and all 60 models' meta now
round-trip; jaffle-shop and demo-f1 likewise.
Lightdash treats every dbt column as a dimension by default; only
meta.dimension.hidden withdraws it from grouping. The converter read the
presence of meta.dimension as the dimension marker instead, so a real
project exported most of its columns as measure-only fields (5,481 of
7,435 on the Lightdash analytics project).
Import now emits dimension: {} for every column that is not hidden, and no
dimension for a hidden one. Export writes a measure-only Ossie field as a
hidden dimension, keeping its type, label and AI hint, and writes nothing
for a dimension with nothing else to say. FIELD_ATTRIBUTE_NOT_REPRESENTABLE
is gone: labels and hints always have a home now.
A metric's filters and a model's sql_filter / sql_where / required_filters travel in the lightdash extension and round-trip, but every other consumer of the document sees the unfiltered aggregate or the unrestricted dataset. Both are now reported on import (METRIC_FILTER_NOT_PORTABLE, ROW_FILTER_NOT_PORTABLE) instead of passing silently.
Add LIGHTDASH to the spec's vendor-name table, the schema examples, the Python OssieVendor enum and the converters README, and make the Lightdash converter emit the registered token. Documents written with the earlier lowercase name are still read.
Restructure the README around usage, the Python API, the mapping, and a section on why each loss is unavoidable (name scope, what a dimension is, joins versus relationships, query-time evaluation, types), followed by what is kept for Lightdash only, approximated, and not carried.
An Ossie document now becomes a deployable Lightdash project without a dbt project around it: export writes lightdash/models/<model>.yml per dataset (type: model, sql_from = the dataset's source, typed dimensions with their own sql, metrics and joins in place) plus a starter lightdash.config.yml whose warehouse type follows --dialect or --warehouse. Model meta the dbt flavour has to stash becomes ordinary top-level keys. --format dbt-meta keeps the single schema.yml output for dbt projects. lightdash compile accepts the exported TPC-DS project: 5 explores, 0 errors.
dbt projects keep one schema YAML per model, so import now accepts a directory as well as a file: it walks it in sorted order, merges every list-valued models: and seeds: entry, and ignores target/, dbt_packages/ and dbt_project.yml. load_schema is exported for the Python API.
The CLI printed nothing on success and bare issue codes. It now reports what it wrote (model files and config, or datasets, metrics and relationships), explains each issue type the first time it appears, and closes with a count. Everything goes to stderr, issues first, like the other converters; stdout stays clean.
Lightdash learns most dimension types from the warehouse, not from YAML, so a Lightdash project imports with most fields untyped and the model-file export has to assume string for them. --catalog reads the catalog.json that dbt docs generate writes and fills datatype for every column without an authored type, reducing the physical type to Ossie's vocabulary (INT64 -> Integer, NUMBER(12,2) -> Decimal, TIMESTAMP_TZ -> DateTimeTz). Authored types win. A model missing from the catalog is reported, which is how a stale catalog shows up. On demo-f1 this takes the model-file export from 78 assumed types to 1.
The positional form still works.
A config written without a warehouse type carried CHANGE_ME and was then never rewritten, so re-running with --dialect fixed the models but left lightdash compile refusing the placeholder. The config is now rewritten while it still holds the placeholder and kept once a real type is in it.
…y element A real project produced hundreds of one-line issues. The report is now one block per issue type: a header with the count, the explanation, and the affected elements wrapped to a readable width (the first few, or all of them with -v/--verbose).
…project A template with placeholders or a Jinja-only file crashed the walk; it is now skipped with a note naming the file. Virtualenv directories (env, venv, .venv, site-packages) are ignored like target/ and dbt_packages/.
import now reads Lightdash model files (type: model, sql_from, a dimensions list) as well as dbt schema files, mixed freely in one directory. A model file is folded into the dbt model shape: dimensions become columns, its sql_from is the dataset source verbatim (so --database/--schema are not needed for it), and the rest becomes model meta. This closes the round trip for dbt-less Lightdash projects and for anything export writes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the bidirectional apache-ossie-lightdash converter and ossie-lightdash CLI.
Related Issues
Closes #227
Supersedes #253 (with permission from @ota2000
Checklist