Add structured file-backed dataset sources - #338
Conversation
Signed-off-by: isaac-dasari <74317395+isaac-dasari@users.noreply.github.com>
|
@isaac-dasari would it possible to fix the conflicts? Thanks! |
Signed-off-by: isaac-dasari <74317395+isaac-dasari@users.noreply.github.com>
|
@jbonofre Thanks the conflicts with current main are resolved and the branch is updated. I also adapted the changes to the completed OSI → Ossie rename and reran the affected Python and Java test suites locally. The PR is mergeable again; the fork-triggered GitHub Actions runs are now pending approval. |
|
@isaac-dasari awesome! Thanks! I will do a new pass. |
| File-backed datasets may use an explicit structured form. `kind` is the discriminator; the initial portable file form requires a physical format and at least one location: | ||
|
|
||
| ```yaml | ||
| source: | ||
| kind: file | ||
| format: parquet | ||
| locations: | ||
| - s3://analytics-data/orders/*.parquet |
There was a problem hiding this comment.
This is a great enhancement to the spec. But how about we generallize this such that it can support any catalog type (Hive, Polaris etc.. ) and then format can be parquet , table or sql query
Examples:
kind: file
format: parquet
locations:
- s3://analytics-data/orders/*.parquet
kind: HIVE_CATALOG
format: table
identifier: db.schema.table
kind: SNOWFLAKE_CATALOG
format: SQL_QUERY
query: 'select * from db.schema.table where col1 >10'
There was a problem hiding this comment.
Thanks for tagging me here. That's a good point. I think 3 catalogs(file/catalog entity/sql) makes sense in general. Have we thought about consolidate SQL_QUERY to catalog entity(e.g., view)?
Signed-off-by: isaac-dasari <74317395+isaac-dasari@users.noreply.github.com>
| # Structured catalog table form: | ||
| # source: | ||
| # kind: HIVE_CATALOG | ||
| # format: table |
There was a problem hiding this comment.
Do we need a format here? What would be the expected values? table, view, or UDF? We may only support table and view. We may not need a format in that case. The identifier for them are unified across multiple systems.
| # | ||
| # Structured SQL query form: | ||
| # source: | ||
| # kind: SNOWFLAKE_CATALOG |
There was a problem hiding this comment.
Yes, that's part of the spec about SQL vendor/dialect.
|
@isaac-dasari thanks for the PR. I think it goes the right direction. As a reminder, per last community sync, we will need a dev mailing discussion for spec changes. Do you mind filing one? |
4 participants
Summary
This draft adds a backward-compatible structured representation for file-backed datasets in
Dataset.sourcewhile preserving all existing string source forms.