[Bug] BE crashed with SIGSEGV and auto-restarted, causing query disruption while it recovered.

[yk-meta]

Search before asking

• I had searched in the issues and found no similar issues.

Version

4.0.7-rc02-35854e7e92a

What's Wrong?

grep -A 15 "1784283889" be.out
*** Aborted at 1784283889 (unix time) try "date -d @1784283889" if you are using GNU date ***
*** Current BE git commitID: 35854e7 ***
*** SIGSEGV address not mapped to object (@0x1f8) received by PID 2929477 (TID 3000255 OR 0x7d4622d7b6c0) from PID 504; stack trace: ***
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_release/doris/be/src/common/signal_handler.h:420
1# PosixSignals::chained_handler(int, siginfo_t*, void*) in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
2# JVM_handle_linux_signal in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
3# 0x00007D6150845330 in /lib/x86_64-linux-gnu/libc.so.6
4# doris::RuntimeState::is_nereids() const at /home/zcp/repo_center/doris_release/doris/be/src/runtime/runtime_state.cpp:511
5# doris::AttachTask::AttachTask(doris::RuntimeState*) at /home/zcp/repo_center/doris_release/doris/be/src/runtime/thread_context.cpp:47
6# std::_Function_handler<void (), doris::vectorized::AsyncResultWriter::start_writer(doris::RuntimeState*, doris::RuntimeProfile*)::$_0>::_M_invoke(std::_Any_data const&) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:292
7# doris::ThreadPool::dispatch_thread() at /home/zcp/repo_center/doris_release/doris/be/src/util/threadpool.cpp:623
8# doris::Thread::supervise_thread(void*) at /home/zcp/repo_center/doris_release/doris/be/src/util/thread.cpp:461
9# 0x00007D615089CAA4 in /lib/x86_64-linux-gnu/libc.so.6
10# 0x00007D6150929C6C in /lib/x86_64-linux-gnu/libc.so.6

INFO: java_cmd /usr/lib/jvm/java-17-openjdk-amd64/bin/java

What You Expected?

No BE crash

How to Reproduce?

Not yet isolated into a minimal repro. Observed in production: BE crashed with the above
trace while running queries against a JDBC External Catalog (MS SQL Server). The crash
occurs inside the async result-writer's thread pool dispatch, when constructing an
AttachTask from a RuntimeState pointer — consistent with the RuntimeState having already
been destroyed by that point (use-after-free)

Anything Else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[yk-meta]

SEVERE: ConnectionID:32909 ClientConnectionId: 2135a909-abd0-441a-80e3-b4c7c505fd1b: getNextResult: Encountered unexpected TDS_FEATURE_EXT_UTF8SUPPORT (0x0A)
Jul 17, 2026 10:24:08 AM com.microsoft.sqlserver.jdbc.TDSReader throwInvalidTDSToken
SEVERE: ConnectionID:32909 ClientConnectionId: 2135a909-abd0-441a-80e3-b4c7c505fd1b got unexpected value in TDS response at offset:7986
Jul 17, 2026 10:24:08 AM com.microsoft.sqlserver.jdbc.TDSParser throwUnexpectedTokenException
SEVERE: ConnectionID:33025 ClientConnectionId: 693a78e3-ca12-4956-a89c-9168c75d4633: getNextResult: Encountered unexpected unknown token (0x6)
Jul 17, 2026 10:24:08 AM com.microsoft.sqlserver.jdbc.TDSReader throwInvalidTDSToken
SEVERE: ConnectionID:33025 ClientConnectionId: 693a78e3-ca12-4956-a89c-9168c75d4633 got unexpected value in TDS response at offset:7989
Jul 17, 2026 10:24:16 AM com.microsoft.sqlserver.jdbc.TDSParser throwUnexpectedTokenException
SEVERE: ConnectionID:32987 ClientConnectionId: a6acf2c3-470f-420c-ae52-798538d6dceb: getNextResult: Encountered unexpected TDS_FEATURE_EXT_UTF8SUPPORT (0x0A)
Jul 17, 2026 10:24:16 AM com.microsoft.sqlserver.jdbc.TDSReader throwInvalidTDSToken
SEVERE: ConnectionID:32987 ClientConnectionId: a6acf2c3-470f-420c-ae52-798538d6dceb got unexpected value in TDS response at offset:7986
Jul 17, 2026 10:24:23 AM com.microsoft.sqlserver.jdbc.TDSParser throwUnexpectedTokenException
SEVERE: ConnectionID:33093 ClientConnectionId: 10c67f25-cf77-42c9-a35f-6d98b8ba057f: getNextResult: Encountered unexpected unknown token (0x30)
Jul 17, 2026 10:24:23 AM com.microsoft.sqlserver.jdbc.TDSReader throwInvalidTDSToken
SEVERE: ConnectionID:33093 ClientConnectionId: 10c67f25-cf77-42c9-a35f-6d98b8ba057f got unexpected value in TDS response at offset:7991
Jul 17, 2026 10:24:23 AM com.microsoft.sqlserver.jdbc.TDSParser throwUnexpectedTokenException
SEVERE: ConnectionID:33094 ClientConnectionId: 54a70d1d-c72a-4dea-b548-ac6b375dc7b1: getNextResult: Encountered unexpected unknown token (0x30)
Jul 17, 2026 10:24:23 AM com.microsoft.sqlserver.jdbc.TDSReader throwInvalidTDSToken
SEVERE: ConnectionID:33094 ClientConnectionId: 54a70d1d-c72a-4dea-b548-ac6b375dc7b1 got unexpected value in TDS response at offset:7991
Jul 17, 2026 10:24:25 AM com.microsoft.sqlserver.jdbc.TDSParser throwUnexpectedTokenException
SEVERE: ConnectionID:33129 ClientConnectionId: 1337689e-a687-45c8-a616-c6187be49383: getNextResult: Encountered unexpected unknown token (0xC)
Jul 17, 2026 10:24:25 AM com.microsoft.sqlserver.jdbc.TDSReader throwInvalidTDSToken
SEVERE: ConnectionID:33129 ClientConnectionId: 1337689e-a687-45c8-a616-c6187be49383 got unexpected value in TDS response at offset:7987
Jul 17, 2026 10:24:28 AM com.microsoft.sqlserver.jdbc.TDSParser throwUnexpectedTokenException
SEVERE: ConnectionID:33172 ClientConnectionId: fb1ee5d8-0784-4bcb-95e0-0fb9fc13679e: getNextResult: Encountered unexpected unknown token (0x6)
Jul 17, 2026 10:24:28 AM com.microsoft.sqlserver.jdbc.TDSReader throwInvalidTDSToken
SEVERE: ConnectionID:33172 ClientConnectionId: fb1ee5d8-0784-4bcb-95e0-0fb9fc13679e got unexpected value in TDS response at offset:7990
*** Query id: 0-0 ***
*** is nereids: 1 ***
*** tablet id: 0 ***
*** Aborted at 1784283889 (unix time) try "date -d @1784283889" if you are using GNU date ***

This is the logs right before crash

[zclllyybb]

Breakwater-GitHub-Analysis-Slot: slot_a2c29ec04503
This content is generated by AI for reference only.

Initial triage:

I checked the reported tag 4.0.7-rc02 / commit 35854e7e92a. This looks like a real lifetime bug in the async writer path, not enough evidence yet for an MS SQL Server JDBC connector-specific root cause.

The crash frame is consistent with the current code order:

1. AsyncResultWriter::start_writer() captures raw RuntimeState* state, raw RuntimeProfile* operator_profile, and a weak task execution context.
2. In the submitted thread-pool callback, it runs SCOPED_ATTACH_TASK(state) first.
3. Only after that does it call task_ctx.lock() and return if the context has expired.
4. SCOPED_ATTACH_TASK(state) enters AttachTask(RuntimeState*), which immediately calls runtime_state->is_nereids().
5. RuntimeState::is_nereids() dereferences _query_ctx.

So if the async writer callback starts after the fragment/task context has already begun teardown, the weak lifetime guard is checked too late. The reported SIGSEGV in RuntimeState::is_nereids() at address @0x1f8 matches that failure mode. I also checked current upstream master: the file has moved to be/src/exec/sink/writer/async_result_writer.cpp, but the same callback order is still present there, so I do not see an already-merged public fix for this exact ordering.

Impact/scope:

• This affects the generic AsyncWriterSink / AsyncResultWriter path. JDBC table sink is one user of it, and file result, tablet, Hive, and Iceberg writers also use the same base pattern.
• Because the crash happens before the concrete writer's open() / write() implementation appears in the stack, the available stack does not prove the MS SQL Server connector itself is faulty.
• If this was a plain SELECT from a JDBC external catalog to the client, the exact SQL/profile is needed to identify which concrete sink was active, since the normal result sink path is different from the async writer sink path.

Suggested next step for a fix:

• In the async writer callback, lock the task execution context before any use of state, operator_profile, or this, especially before SCOPED_ATTACH_TASK(state).
• Keep the strong task context lock alive through process_block().
• If the lock fails, exit without touching the captured raw pointers, and make sure the writer status / finish dependency cannot leave the pipeline waiting in any still-live close path.
• Add a focused regression/unit test or debug-only test where the async writer task is submitted but the task context expires before the queued callback starts; the callback should return without dereferencing RuntimeState.

Missing information that would help confirm the trigger and choose the safest patch:

• Full be.INFO / be.out around the crash timestamp, especially query id, fragment instance id, cancellation, timeout, and client disconnect logs.
• The query profile or profile id for the failed query.
• The exact SQL shape, sanitized if needed: plain SELECT, INSERT INTO JDBC table, CTAS, OUTFILE/export, or another sink.
• Whether the query was cancelled, timed out, or the client disconnected around the crash time.
• If available, the coredump backtrace with local variables for the async writer callback frame, especially the captured state, task_ctx, and task/fragment context state.

[zclllyybb]

Breakwater-GitHub-Analysis-Slot: slot_a2c29ec04503
This content is generated by AI for reference only.

Follow-up after the additional log comment:

The newly added JDBC logs change the trigger picture, but not the BE crash mechanism from the stack.

What the new logs add:

• The Microsoft SQL Server JDBC driver reported repeated TDS parsing failures between 2026-07-17 10:24:08 UTC and 2026-07-17 10:24:28 UTC.
• The BE abort timestamp is 2026-07-17 10:24:49 UTC, so those JDBC errors happened immediately before the crash.
• In the Doris 4.0.7 code path, JdbcScanner::_open_impl() / _get_block_impl() call JdbcConnector::query() / get_next(), and JNI/JDBC exceptions are supposed to be returned as Status errors. Such a JDBC-side failure should fail/cancel the query, not crash the BE process.

Updated interpretation:

• There are likely two related issues in this incident:
  1. The query first hit a SQL Server JDBC/TDS protocol/driver-side error while reading from the external catalog.
  2. The BE process then crashed in the async writer cleanup/teardown path because AsyncResultWriter::start_writer() checks the weak task context only after SCOPED_ATTACH_TASK(state) has already dereferenced RuntimeState.
• The added signal line Query id: 0-0 also fits this: the async writer thread had not successfully attached a task id before the signal. The is nereids: 1 line alone should not be overinterpreted, because that thread-local flag is not reset in AttachTask::~AttachTask() in this branch and may be stale from a reused thread-pool thread.

For the BE crash, I still recommend fixing the async writer lifetime ordering first:

• Lock the task execution context at the very beginning of the submitted callback.
• Do not touch state, operator_profile, or this before that lock succeeds.
• Keep the strong task context lock alive through process_block().
• Make the early-return path also settle writer status / finish dependency if any still-live close path can wait on it.

For the preceding SQL Server/JDBC trigger, the next useful evidence is:

• The exact SQL shape and whether the plan uses plain JDBC scan, INSERT INTO ... SELECT, CTAS, OUTFILE/file result, or another async writer sink.
• The JDBC catalog properties with secrets removed: driver_url, driver_class, SQL Server JDBC driver jar/version, JDBC URL options such as encrypt / trustServerCertificate, and whether force_sqlserver_jdbc_encrypt_false is enabled.
• SQL Server version/edition and whether this is Azure SQL or a proxy/gateway endpoint.
• The query profile or FE audit/query id that maps the JDBC exceptions to the crashing BE fragment.

If the TDS errors disappear after changing the SQL Server JDBC driver or connection options, that would explain the trigger, but the BE should still not crash on that error path; the async writer lifetime fix is still needed.