MSSQL Source Connector by lsabi · Pull Request #3851 · apache/iggy

lsabi · 2026-08-09T19:56:53Z

Which issue does this PR address?

Partially Closes # #3573
Relates to #

Rationale

This is just the implementation of the source connector for the cdc. I'm also working on the table polling, but it'll require more time.

I don't have a database instance to test it, but tests are passing. If someone, who has a test MSSQL instance can try it out, it would be great.

What changed?

Added the

Local Execution

Unit Tests pass. Not tested with a real database instance, as I don't have an instance with dummy data to ruin (only production)
Pre-commit hooks not ran

AI Usage

None

github-actions · 2026-08-09T19:57:01Z

Thanks for the PR. It is labeled S-waiting-on-review and queued for review.

Slash commands (own line, regular comment) move it around the queue:

/ready - back to S-waiting-on-review after addressing feedback
/author - flip to S-waiting-on-author while you finish changes
/request-review @user-or-team - request a reviewer

See CONTRIBUTING.md for details.

lukaszzborek · 2026-08-09T20:14:57Z

@lsabi please make this pr as draft or close it and add test, then reopen
there is docker image microsoft/mssql-server which you can use

codecov · 2026-08-10T08:24:27Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 80.36%. Comparing base (e71e3c6) to head (a33163b).
⚠️ Report is 3 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #3851      +/-   ##
============================================
- Coverage     83.67%   80.36%   -3.31%     
+ Complexity     1358     1340      -18     
============================================
  Files          1212     1218       +6     
  Lines        165138   164790     -348     
  Branches     132612   132723     +111     
============================================
- Hits         138181   132439    -5742     
- Misses        23308    28803    +5495     
+ Partials       3649     3548     -101

Components	Coverage Δ
Rust Core	`83.81% <ø> (-0.63%)`	⬇️
Java SDK	`66.54% <ø> (-0.13%)`	⬇️
C# SDK	`40.03% <ø> (-36.06%)`	⬇️
Python SDK	`90.00% <ø> (ø)`
PHP SDK	`84.48% <ø> (ø)`
Node SDK	`89.09% <ø> (-6.76%)`	⬇️
Go SDK	`54.54% <ø> (-13.79%)`	⬇️
see 284 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

lsabi · 2026-08-12T13:52:59Z

@lukaszzborek I'm afraid but I can't test it on my locale. I've spent a day understanding the docs, as it's not super clear how one should do in order to build their own connector.

I've then let my laptop run for almost a day and it did not finish compiling, just excessive heating.

I managed to compile mssql_source in a couple of hours, but not the runtime nor even the whole project. If you want, you can test it, but I've discovered that iggy and runtime are separate and require two different processes in order to run, which is what I did not want (I though the dynamic runtime could start up libraries on the fly and was part of the "core").

lukaszzborek · 2026-08-13T22:04:22Z

@lsabi, quite intresting. Did you join to our discord? there are many people which can help. You can find discord invite in readme. Also you can merge current master to yours, because you have old version of server. And please write on discord what machine do you have (where you compile code)

hubcio

took this for a spin against a real sql server 2022 container with cdc enabled. open() connects fine, but as shipped poll() never returns a single message - it took five local fixes to get the first record through, and the payloads were still mangled. detailed comments inline.

hubcio · 2026-08-14T13:18:02Z

+				FROM [{db}].cdc.[{table}] t
+				WHERE {query_filter}
+				GROUP BY [__$start_lsn], [__$command_id], [__$seqval], [__$end_lsn], [__$update_mask]
+				ORDER BY [__$start_lsn] ASC, [__$command_id] ASC;";


leftover ;"; inside the sql string - every poll fails with error 105 "unclosed quotation mark after the character string". ran the connector against sql server 2022 with cdc enabled: as shipped it never produces a single message.

hubcio · 2026-08-14T13:18:02Z

+			.flatten()
+			.collect::<Vec<_>>();
+
+		let tables: Vec<String> = self.config.capture_table_columns.values().cloned().collect();


this iterates the map's values (the column lists) as table names, so the query becomes FROM [testdb].cdc.[id,name,active] and fails with error 208 "invalid object name".

hubcio · 2026-08-14T13:18:02Z

+		// For each monitored table, get the capture columns
+		let table_columns = self.query(Query::new(
+			format!(r#"
+				SELECT OBJECT_NAME(ct.object_id, DB_ID('{database}')) as table_name, c.object_id, ct.capture_instance, ct.start_lsn, c.columns FROM [{database}].[cdc].[change_tables] as ct JOIN (SELECT object_id, STRING_AGG(column_name, ',') as columns FROM [{database}].[cdc].[captured_columns] GROUP BY object_id) as c ON c.object_id = ct.object_id;"#


OBJECT_NAME(ct.object_id, DB_ID(...)) names the change table (dbo_users_CT), not the source table - in cdc.change_tables, object_id is the change table's id and source_object_id is the source table's. so capture_table_columns ends up keyed by change-table names and a user-provided entry never matches the lookup on line 411. use ct.source_object_id.

Since MSSQL has the source table and the change table with different names (by default _ct get's appended), either the user inserts the change table name or, there's need to make an internal conversion (while keeping a mapping of the names). Which one is the preferred one?

hubcio · 2026-08-14T13:18:02Z

+			let changes_query = format!(
+				r#"SELECT 
+						TODATETIMEOFFSET(
+						[#db].sys.fn_cdc_map_lsn_to_time([__$start_lsn]),


#db is only substituted in the max-lsn query; here it reaches the server literally and fails with error 4121. the surrounding query already interpolates {db} via format!.

hubcio · 2026-08-14T13:18:03Z

+					None => return Err(Error::InvalidRecord)
+				};
+
+				let operation_type = match r.get::<i64, _>("[__$operation]").unwrap_or(0) {


this panics instead of erroring: the result column is __$operation (no brackets) and it's an int, so both the name and the i64 type are wrong. tiberius get is try_get(..).unwrap(), and try_get errors on an unknown column name as well as on a failed conversion - so unwrap_or(0) is unreachable. the panic lands inside the task the sdk spawns for poll(), killing the polling task silently: the connector still reports as running and never produces again. use try_get with i32 and return an error.

hubcio · 2026-08-14T13:18:03Z

+					self.id
+				);
+			}
+			"polling" => {


"polling" is accepted here but poll() returns Err(InvalidConfig) forever since poll_tables is commented out - reject the mode in open() until it exists.

hubcio · 2026-08-14T13:18:03Z

+			format!("[__$start_lsn] <= {n_lsn}")
+		};
+
+		let capture_ops = self


capture_ops is built but never used - the capture_operations filter is silently ignored and every operation type gets streamed.

hubcio · 2026-08-14T13:18:03Z

+	}
+
+
+	/*


over half the file is commented-out postgres source code (poll_tables, sqlx type mapping, nearly all tests) - please delete it, git keeps history. the crate also fails cargo fmt --check (tabs) and cargo clippy -- -D warnings (21 errors), both are ci gates.

hubcio · 2026-08-14T13:18:03Z

+
+[dependencies]
+async-trait = { workspace = true }
+base64 = { workspace = true }


base64 is only referenced inside commented-out code - dead dependency, drop it. machete won't catch this one: its heuristic matches the crate name inside comments, so the dead code keeps it looking used.

hubcio · 2026-08-14T13:18:03Z

    "core/connectors/sources/elasticsearch_source",
    "core/connectors/sources/influxdb_source",
    "core/connectors/sources/postgres_source",
+    "core/connectors/sources/mssql_source",


Cargo.lock isn't part of the PR - building adds ~238 lines (tiberius, async-native-tls, connection-string, ...). commit the lock update.

lsabi added 7 commits August 3, 2026 22:57

Update Cargo.toml

9d95996

Create lib.rs

ec12805

Create Cargo.toml

2ba76a8

Update Cargo.toml

b317f57

Create Cargo.toml

aa4edc1

Delete core/connectors/sources/mssql_source/src/Cargo.toml

2fdbc8b

Merge branch 'apache:master' into lsabi/mssql-source

3c54da0

github-actions Bot added the S-waiting-on-review PR is waiting on a reviewer label Aug 9, 2026

lukaszzborek added S-waiting-on-author PR is waiting on author response and removed S-waiting-on-review PR is waiting on a reviewer labels Aug 9, 2026

lsabi marked this pull request as draft August 10, 2026 12:06

github-actions Bot removed the S-waiting-on-author PR is waiting on author response label Aug 10, 2026

Merge branch 'apache:master' into lsabi/mssql-source

cceda10

hubcio requested changes Aug 14, 2026

View reviewed changes

lsabi added 11 commits August 15, 2026 10:05

Merge branch 'apache:master' into lsabi/mssql-source

e4ae295

Merge branch 'apache:master' into lsabi/mssql-source

9cce184

Merge branch 'apache:master' into lsabi/mssql-source

62f8f32

Fix comments from PR apache#3851

21c6097

🔥 Remove unused dependencies

f3982ff

Set default mandatory values in config.toml

2244d52

Delete unused dependencies

a22db22

Merge branch 'apache:master' into lsabi/mssql-source

66c4ab9

Add mssql_server feature for test containers

4658b2c

Add test init files for mssql

267f011

Add test

d27db8e

lsabi added 11 commits August 18, 2026 19:16

Add test init files for mssql

7ea9743

Add mssql module

9acd5c2

Add test init files for mssql

cfaa83f

Add test init files for mssql

865f11f

Add test init files for mssql

c89534c

Add test init files for mssql

28c9afe

Add test init files for mssql

8226a49

Add db connection function

453635c

Delete file added by mistake

a2fbcbe

Delete file added by mistake

70336c5

Add test for mssql connector

a33163b

Uh oh!

Conversation

lsabi commented Aug 9, 2026

Which issue does this PR address?

Rationale

What changed?

Local Execution

AI Usage

Uh oh!

github-actions Bot commented Aug 9, 2026

Uh oh!

lukaszzborek commented Aug 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented Aug 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

lsabi commented Aug 12, 2026

Uh oh!

lukaszzborek commented Aug 13, 2026

Uh oh!

hubcio left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

lukaszzborek commented Aug 9, 2026 •

edited

Loading

codecov Bot commented Aug 10, 2026 •

edited

Loading