Skip to content

feat(connectors): Clickhouse Sink Connector#2886

Open
kriti-sc wants to merge 54 commits into
apache:masterfrom
kriti-sc:clickhouse-sink
Open

feat(connectors): Clickhouse Sink Connector#2886
kriti-sc wants to merge 54 commits into
apache:masterfrom
kriti-sc:clickhouse-sink

Conversation

@kriti-sc
Copy link
Copy Markdown
Contributor

@kriti-sc kriti-sc commented Mar 6, 2026

Which issue does this PR close?

Closes #2539

Rationale

Clickhouse is a real-time data analytics engine, and very popular in modern analytics architectures.

What changed?

This PR introduces a Clickhouse Sink Connector that enables writing data from Iggy to Clickhouse.

The Clickhouse writing logic is heavily inspired by the official Clickhouse Kafka Connector.

Local Execution

  • Produced messages 30456 + 29060 rows with schema user_id: String, user_type: u8, email: String, source: String, state: String, created_at: DateTime, message: String using sample data producer.
  • Consumed messages using the Clickhouse sink and into the particular Clickhouse table.
  • Verified schema and number of rows in Clickhouse.
  • Added unit tests and e2e tests, both passing.

Images 1&2: Produced 30456 + 29060 rows into Iggy in two batches
Image 3: Verified schema and number of rows in Clickhouse

image image image

AI Usage

  1. Which tools? (e.g., GitHub Copilot, Claude, ChatGPT) Claude Code
  2. Scope of usage? (e.g., autocomplete, generated functions, entire implementation) generated functions
  3. How did you verify the generated code works correctly? Manual testing by producing data into Iggy and then running the sink and verifying insertion into Clickhouse, unit tests and e2e tests for different Clickhouse insert configurations.
  4. Can you explain every line of the code if asked? Yes

@codecov
Copy link
Copy Markdown

codecov Bot commented Mar 6, 2026

Codecov Report

❌ Patch coverage is 82.56210% with 358 lines in your changes missing coverage. Please review.
✅ Project coverage is 48.11%. Comparing base (7bbe10c) to head (ec1064f).

Files with missing lines Patch % Lines
...ore/connectors/sinks/clickhouse_sink/src/binary.rs 82.95% 122 Missing and 56 partials ⚠️
...ore/connectors/sinks/clickhouse_sink/src/schema.rs 76.45% 30 Missing and 75 partials ⚠️
...ore/connectors/sinks/clickhouse_sink/src/client.rs 72.68% 55 Missing and 10 partials ⚠️
core/connectors/sinks/clickhouse_sink/src/lib.rs 97.27% 3 Missing and 1 partial ⚠️
core/connectors/sinks/clickhouse_sink/src/sink.rs 60.00% 4 Missing ⚠️
core/connectors/sinks/clickhouse_sink/src/body.rs 98.80% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@              Coverage Diff              @@
##             master    #2886       +/-   ##
=============================================
- Coverage     74.44%   48.11%   -26.33%     
  Complexity      943      943               
=============================================
  Files          1231     1235        +4     
  Lines        120911   107676    -13235     
  Branches      97647    84440    -13207     
=============================================
- Hits          90017    51813    -38204     
- Misses        27943    53043    +25100     
+ Partials       2951     2820      -131     
Components Coverage Δ
Rust Core 41.83% <82.56%> (-33.76%) ⬇️
Java SDK 58.44% <ø> (ø)
C# SDK 69.41% <ø> (-0.52%) ⬇️
Python SDK 81.06% <ø> (ø)
Node SDK 91.44% <ø> (-0.10%) ⬇️
Go SDK 40.20% <ø> (ø)
Files with missing lines Coverage Δ
core/connectors/sinks/clickhouse_sink/src/body.rs 98.80% <98.80%> (ø)
core/connectors/sinks/clickhouse_sink/src/lib.rs 97.27% <97.27%> (ø)
core/connectors/sinks/clickhouse_sink/src/sink.rs 60.00% <60.00%> (ø)
...ore/connectors/sinks/clickhouse_sink/src/client.rs 72.68% <72.68%> (ø)
...ore/connectors/sinks/clickhouse_sink/src/schema.rs 76.45% <76.45%> (ø)
...ore/connectors/sinks/clickhouse_sink/src/binary.rs 82.95% <82.95%> (ø)

... and 351 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Comment thread core/connectors/sinks/clickhouse_sink/src/binary.rs
@kriti-sc kriti-sc requested a review from abonander March 7, 2026 10:58
Comment thread core/connectors/sinks/clickhouse_sink/src/binary.rs Outdated
@github-actions
Copy link
Copy Markdown

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs.

If you need a review, please ensure CI is green and the PR is rebased on the latest master. Don't hesitate to ping the maintainers - either @core on Discord or by mentioning them directly here on the PR.

Thank you for your contribution!

@github-actions github-actions Bot added S-stale Inactive issue or pull request and removed S-stale Inactive issue or pull request labels Mar 17, 2026
Copy link
Copy Markdown
Contributor

@hubcio hubcio left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

overall good direction, just needs a little bit polishing

Comment thread core/connectors/sinks/clickhouse_sink/src/binary.rs Outdated
Comment thread core/connectors/sinks/clickhouse_sink/src/binary.rs Outdated
Comment thread core/connectors/sinks/clickhouse_sink/src/schema.rs Outdated
Comment thread core/connectors/sinks/clickhouse_sink/src/binary.rs Outdated
Comment thread core/connectors/sinks/clickhouse_sink/src/binary.rs Outdated
Comment thread core/connectors/sinks/clickhouse_sink/src/binary.rs Outdated
Comment thread core/connectors/sinks/clickhouse_sink/src/client.rs
Comment thread core/connectors/sinks/clickhouse_sink/src/binary.rs Outdated
Comment thread core/connectors/sinks/clickhouse_sink/src/binary.rs Outdated
Comment thread core/connectors/sinks/clickhouse_sink/src/binary.rs Outdated
@github-actions
Copy link
Copy Markdown

github-actions Bot commented Apr 2, 2026

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs.

If you need a review, please ensure CI is green and the PR is rebased on the latest master. Don't hesitate to ping the maintainers - either @core on Discord or by mentioning them directly here on the PR.

Thank you for your contribution!

@github-actions github-actions Bot added S-stale Inactive issue or pull request and removed S-stale Inactive issue or pull request labels Apr 2, 2026
@kriti-sc kriti-sc requested a review from hubcio April 3, 2026 19:56
@github-actions
Copy link
Copy Markdown

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs.

If you need a review, please ensure CI is green and the PR is rebased on the latest master. Don't hesitate to ping the maintainers - either @core on Discord or by mentioning them directly here on the PR.

Thank you for your contribution!

@github-actions github-actions Bot added S-stale Inactive issue or pull request and removed S-stale Inactive issue or pull request labels Apr 18, 2026
@kriti-sc
Copy link
Copy Markdown
Contributor Author

kriti-sc commented Jun 2, 2026

/ready

@github-actions github-actions Bot added S-waiting-on-review PR is waiting on a reviewer and removed S-waiting-on-author PR is waiting on author response labels Jun 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

S-waiting-on-review PR is waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement ClickHouse Sink Connector

5 participants