Skip to content

Implement UDP transport for air-gap sink - #17954

Merged
jt2594838 merged 14 commits into
masterfrom
air-gap-udp-transport
Oct 30, 2026
Merged

jt2594838 merged 14 commits into
masterfrom
air-gap-udp-transport

Conversation

@Caideyipi

Copy link
Copy Markdown
Collaborator

Summary

  • Add tcp/udp transport option and UDP packet size validation for air-gap connector/sink.
  • Send UDP air-gap datagrams with the existing envelope/ACK flow and slice oversized requests.
  • Make the receiver listen on UDP on the air-gap port and keep per UDP client receiver/session state.

Tests

  • mvn -DskipTests spotless:apply -pl iotdb-core/node-commons,iotdb-core/datanode
  • mvn -pl iotdb-core/node-commons -DskipTests compile
  • git diff --check
  • mvn -pl iotdb-core/datanode -Dtest=IoTDBDataRegionAirGapSinkTest,IoTDBAirGapReceiverTest test (fails during existing datanode compilation: missing parser/generated/calc/consensus symbols such as AlterTopicContext, ComparatorChain, PERCENTILE, and IStateMachine.loadSnapshot return mismatch)

@codecov

codecov Bot commented Jun 16, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 66.66667% with 106 lines in your changes missing coverage. Please review.
✅ Project coverage is 47.99%. Comparing base (8effe5d) to head (39dcb47).
⚠️ Report is 131 commits behind head on master.

Files with missing lines Patch % Lines
...iver/protocol/airgap/IoTDBAirGapReceiverAgent.java 23.07% 70 Missing ⚠️
...db/commons/pipe/sink/protocol/IoTDBAirGapSink.java 85.13% 22 Missing ⚠️
.../receiver/protocol/airgap/IoTDBAirGapReceiver.java 80.00% 14 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master   #17954      +/-   ##
============================================
+ Coverage     45.95%   47.99%   +2.03%     
+ Complexity      712      604     -108     
============================================
  Files          5505     5572      +67     
  Lines        400063   409989    +9926     
  Branches      52185    54052    +1867     
============================================
+ Hits         183868   196784   +12916     
+ Misses       216195   213205    -2990     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

# Conflicts:
#	iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/pipe/sink/protocol/IoTDBAirGapSink.java
…-udp-transport

# Conflicts:
#	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/receiver/protocol/airgap/IoTDBAirGapReceiver.java
#	iotdb-core/datanode/src/test/java/org/apache/iotdb/db/pipe/receiver/protocol/airgap/IoTDBAirGapReceiverTest.java
@sonarqubecloud

Copy link
Copy Markdown

Preserve the AirGap retry body fix for both TCP and UDP receiver paths.
…-17954-conflict

# Conflicts:
#	iotdb-core/datanode/src/test/java/org/apache/iotdb/db/pipe/sink/protocol/airgap/IoTDBDataRegionAirGapSinkTest.java
@hongzhi-gao

Copy link
Copy Markdown
Contributor

Automated preliminary correctness review (Codex CLI, based on PR head 4903676cbd2b). No tests were run.

  1. iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/pipe/sink/protocol/IoTDBAirGapSink.java:508

    • Trigger: UDP duplicates an OK acknowledgement for request A. After consuming the first copy, the sender sends request B, which is lost, and receives the duplicate acknowledgement.
    • Potential issue: The response contains only a status byte, and sendOneDatagram accepts it without correlating it with the current request. Consequently, B returns success although the receiver never received it. Serial sending and a connected DatagramSocket do not prevent duplicated or delayed datagrams from the same peer. Requests and acknowledgements need matching identifiers to prevent false success and data loss.
  2. iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/receiver/protocol/airgap/IoTDBAirGapReceiverAgent.java:118

    • Trigger: UDP senders reconnect using new source ports, or datagrams arrive from many distinct endpoints while the service remains running.
    • Potential issue: Each endpoint permanently adds a ClientSession to udpClientSessions before packet validation. Entries are removed only by stopping the entire service; removing the current thread's session after a packet does not remove this map entry. Thus even malformed packets accumulate retained sessions without a bound or expiry. Receiver state created for valid clients likewise has no endpoint retirement path here.

This review is AI-assisted and not an approval or request for changes. Please verify each scenario before acting.

@Caideyipi

Copy link
Copy Markdown
Collaborator Author

@hongzhi-gao Thanks for the review. Both points are addressed in d941282: UDP ACKs now carry request IDs that the sender validates, preventing stale ACKs from acknowledging later requests. UDP sessions are registered only after frame and checksum validation, expire after inactivity, and are capped at 1,024 with receiver cleanup on eviction. The targeted AirGap tests pass (17 tests, 0 failures). Full DataNode test compilation is still blocked by the existing unrelated PipeHeartbeatEvent constructor mismatch in PipeMemoryManagerResizeTest.java:506.

@sonarqubecloud

Copy link
Copy Markdown

Quality Gate Failed Quality Gate failed

Failed conditions
C Security Rating on New Code (required ≥ A)
E Reliability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

@jt2594838
jt2594838 merged commit 574d5eb into master Oct 30, 2026
41 of 42 checks passed
@jt2594838
jt2594838 deleted the air-gap-udp-transport branch October 30, 2026 01:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants