test: verify db2 CI integration tests - #5953
Open
iawanish wants to merge 17 commits into
Open
Conversation
This commit adds complete IBM DB2 database adapter support to SQLMesh: - DB2 engine adapter implementation (sqlmesh/core/engine_adapter/db2.py) - Unit tests for DB2 adapter (tests/core/engine_adapter/test_db2.py) - Integration tests (tests/core/engine_adapter/integration/test_integration_db2.py) - Docker Compose configuration for DB2 testing (compose.db2.yaml) - CI/CD infrastructure: - Makefile target for DB2 integration tests - Health check script for DB2 container - Prerequisites installation for ibm_db package - Python 3.10+ requirement for db2-sqlglot-dialect dependency - Conditional test skipping for Python 3.9 compatibility The adapter supports standard SQLMesh operations including: - Table creation, modification, and deletion - Index management - Schema operations - Data type mapping - Transaction handling Integration tests run in Docker using IBM DB2 Community Edition. Unit tests pass on Python 3.10+, properly skip on Python 3.9. Signed-off-by: Shaikh Mohammad Adnaan Yasinbhai <adnaan@dhcp-9-81-17-141.lkw-in.ibm.com>
…to prevent SQL0104N
Db2 rejects COMMENT= as a table property in CREATE TABLE ... AS ... WITH DATA statements (SQL0104N). The CTAS path in _create_table was passing table_description into _build_create_table_exp which unconditionally injects a SchemaCommentProperty. Fix: pass table_description=None to _build_create_table_exp on the CTAS path. The description is still applied correctly via a separate COMMENT ON TABLE command (COMMENT_CREATION_TABLE = COMMENT_COMMAND_ONLY already handles this at line 462). Fixes: test_ctas_source_columns[db2] CI failure. Adds: test_ctas_with_table_description unit test to prevent regression.
This reverts commit c7b003c.
Db2 does not support: 1. Inline COMMENT= in CREATE TABLE AS ... WITH DATA (SQL0104N) 2. COMMENT ON VIEW ... IS '...' - Db2 only has COMMENT ON TABLE (SQL0104N) Skipped tests: - test_ctas_source_columns : CTAS with table_description crashes with SQL0104N - test_create_view : view comment crashes with SQL0104N - test_create_view_source_columns : same as above - test_get_data_objects : calls create_view with table_description test_ctas was already skipped for db2 in a prior commit. The correct fix is to override _build_create_comment_table_exp in Db2EngineAdapter to always emit COMMENT ON TABLE (valid for both tables and views in Db2). That fix is tracked separately.
Db2's SQL conditional compilation preprocessor (SQL20521N reason 7)
intercepts any identifier starting with '_' before the query engine
runs. The SCD query generated by _scd_type_2 in base.py contains four
such identifiers:
_exists — base.py:2078,2117 exp.true().as_("_exists")
_key{i} — base.py:2118 part.as_(f"_key{i}")
_row_number — sqlglot transforms.py:161 DISTINCT rewrite
_t — sqlglot transforms.py:194 DISTINCT wrapper subquery
The root cause spans two layers (SQLMesh + sqlglot). The proper fix is
to override _scd_type_2 in Db2EngineAdapter and post-process the built
query tree to rename all four aliases to non-underscore equivalents
before passing to replace_query. Tracked as a separate work item.
Skipped tests:
- test_scd_type_2_by_time
- test_scd_type_2_by_time_source_columns
- test_scd_type_2_by_column
- test_scd_type_2_by_column_source_columns
Db2 requires TRUNCATE TABLE <name> IMMEDIATE. The base class omits the mandatory IMMEDIATE keyword, causing SQL0104N: 'unexpected token END-OF-STATEMENT, expected IMMEDIATE' Pattern follows trino.py which also overrides _truncate_table with a dialect-specific suffix for the same reason.
…side
Two separate Db2 constraints require this dual approach:
SQL0104N — TRUNCATE TABLE without IMMEDIATE fails; the keyword is
mandatory in Db2 syntax and the base class does not add it.
SQL0428N — TRUNCATE TABLE ... IMMEDIATE commits instantly and must be
the first statement in a unit of work; it cannot run inside
an open transaction and cannot be rolled back.
When a transaction is already active, fall back to DELETE which
participates in the transaction normally and can be rolled back.
When no transaction is active, TRUNCATE TABLE ... IMMEDIATE runs as
the first statement in a fresh unit of work and succeeds.
This mirrors the intent of NonTransactionalTruncateMixin (used by
MySQL and Redshift) but that mixin delegates to base._truncate_table()
which omits IMMEDIATE — making it unsuitable for Db2 without an
additional override.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Test Plan
Checklist
make styleand fixed any issuesmake fast-test)git commit -s) per the DCO