[fix](remote-doris) Run a remote Doris scan's query when the coordinator dispatches the plan, not while planning

[morningman]

What problem does this PR solve?

Issue Number: None

Related PR: #68338 (this is its follow-up), #68101 / #68266 (why a leaked Flight SQL session costs the catalog user's connection quota on the remote FE), #68743 (the same planning-time execution problem in the ADBC catalog)

Problem Summary:

In short. A remote Doris scan ran its query on the remote cluster, and opened the Flight SQL session it runs in, while the local plan was being translated - before any coordinator existed. Every path that plans without running then needed a patch of its own to end that session, and some had none. A few could not read remote Doris at all (jobs, streaming jobs), and some ran the remote query for nothing (EXPLAIN with a leading comment, the INSERT OVERWRITE probe plan, CREATE JOB). This PR makes the scan run its query only when the coordinator dispatches the plan. The query runs when the coordinator starts the scan's split assignment in exec(), and that split assignment owns the session until the coordinator stops it on close or cancel. A plan that never runs never reaches the remote cluster, and the session has exactly one owner. The first version of this PR patched three more statement owners; that approach is replaced by this one.

The remote Doris change builds on the split assignment of the batch-mode hive / iceberg / paimon / MaxCompute scans, and fixes five things there on the way: a same-plan retry that could only fail (item 2), a 100 ms wait at the end of every backend's fetches (item 3), a backend's split fetch on a connection the frontend had closed, which failed the query right after a frontend restart (item 4), a dropped plan's split generation that held a shared frontend thread forever (item 6), and an unbounded split queue (item 7). Two regression-framework fixes (items 8 and 9) and an unrelated ADBC crash fix (item 10, which can be picked on its own) ride along.

Background. A type = doris catalog with use_arrow_flight = true reads another Doris cluster over Arrow Flight SQL. RemoteDorisScanNode opens a Flight SQL session on a remote FE (the handshake), runs a query there (GetFlightInfo), and hands the endpoints of its result to the local BEs, which read them from the remote BEs with DoGet. The session is a connection of the catalog user on the remote FE. It counts against max_user_connections there, and only CloseSession, KILL or wait_timeout (8h) ends it. It cannot be closed as soon as GetFlightInfo returns, because the remote FE cancels what a closed session still runs. #68338 therefore made the scan hold the session until the coordinator stops it. Because the session was opened while planning, it also registered the scan with the StatementContext as a fallback owner for plans that never get a coordinator.

The problem, and what it cost

Path	Before this PR	After
EXPLAIN SELECT ...	skipped by matching the statement text against "explain"	never dispatched, never reaches the remote FE
/* comment */ EXPLAIN SELECT ...	the match fails: the remote query runs, and the fallback closes its session	never reaches the remote FE
multi-statement packet that cannot be split, EXPLAIN ...; SELECT ...	the SELECT is taken for an EXPLAIN and returns no rows	runs normally
job insert task, streaming insert task	planned with no statement text: NullPointerException in the EXPLAIN match, cannot read remote Doris at all	reads normally
statement refused after planning (SQL block rule on the scan, WITH LABEL already used, txn limit)	remote query already ran; the fallback closes the session	remote query never runs
INSERT OVERWRITE, MTMV refresh	remote query runs twice (the probe plan, then the insert)	once
CREATE JOB (validation plan), streaming task before() (TVF rewrite plan)	remote query runs for a plan that is never executed	never runs
direct COM_STMT_EXECUTE, AutoCloseConnectContext (EXPORT, ANALYZE), streaming task, http_stream refused after planning	no owner ends the statement: session left until wait_timeout	nothing opened before dispatch
query waiting in the workload group queue	holds the remote session and the remote result buffers while it waits	remote query starts after admission
same-plan retry (handleQueryWithRetry)	refused by a remote-Doris-only rule; a batch-mode external scan retried and failed with Split source N is released	one rule for every split-source scan: rethrows the original error
remote frontend that accepts the connection but never answers the handshake	the statement waits for good while it is planned; KILL does not end it	the handshake gives up after the catalog's query_timeout_sec, and the next remote frontend is tried
batch-mode external scan right after the frontend running it restarted	a backend fetches its splits on a cached connection that frontend closed: the query fails with Failed to get batch of split source	the backend connects anew

How this PR fixes it

1. [refactor](fe) A split assignment owns what a batch scan holds for the backends, from dispatch to coordinator close.
   • SplitAssignment gets a start/stop pair. start() registers the split sources with SplitSourceManager and starts the generator. stop() stops the generator, unregisters the sources, and closes what was handed over through addCloseable(); it rethrows a generation failure only after all of that is released.
   • A resource handed over after stop() is closed at once, and a stopped assignment does not start.
   • ScanNode.start() / stop() drive the assignment. Both coordinators call ScanNode.startAll in exec(), once the query is admitted and before the fragments are sent, and ScanNode.stopAll on close() / cancel(). stopAll keeps going past a node that fails to stop.
   • FileQueryScanNode.needsSampleSplit() defaults to true. A scan that plans with its first split (its location type sets the scan params) still starts generating while planning - item 6 makes the statement own that generation until a coordinator takes it over. A scan that answers false generates nothing until dispatch; its startSplit has to queue its splits before it returns, since start() waits only until the first split is published.
2. [fix](fe) ScanNode.cannotBeRedispatched() becomes a general rule: a stopped split assignment cannot serve another dispatch. The same-plan retry of a batch-mode external scan now reports the original error instead of Split source N is released.
3. [improvement](fe) A backend's split fetch no longer waits 100 ms on its queue once the generator has finished: it takes what is queued and returns. Without this, every remote Doris query would wait 100 ms before the first DoGet of each backend holding an endpoint (measured on a one-row lookup, 30 runs right after an FE restart: median 160 ms without it, 58 ms with it); hive / iceberg batch scans lose the same 100 ms tail per backend.
4. [fix](be) A backend no longer takes a cached thrift connection to a server that closed it. Once a frontend restarts, every connection a backend had cached to it is dead. The split fetch (fetchSplitBatch) took one from the cache and failed the scan: unlike the other backend-to-frontend calls it does not reopen and send again, and it cannot, because the frontend hands out the splits of a batch as it answers, so a request sent again after a lost answer would skip a batch. The client cache now looks at a cached client before it hands it out - a non-blocking peek at its socket: the server's end of stream, or an error, is waiting - and closes it instead of handing it out, so every caller gets a live connection. Without this, every remote Doris query, whose backends all fetch after this PR, would fail right after its frontend restarts, as batch-mode hive / iceberg scans already could.
5. [fix](remote-doris) The remote Doris scan becomes a batch split-source scan with needsSampleSplit() == false.
   • Planning only builds the remote query (convertPredicate; EXPLAIN shows it unchanged) and one scan range per backend pointing at a split source.
   • startSplit() runs at dispatch. It tries the remote FEs in turn as before, stopping early if the scan was stopped meanwhile. It hands the session to the split assignment and queues the endpoints as splits.
   • The handshake that opens the session gets a deadline, the catalog's query_timeout_sec, as GetFlightInfo has. The coordinator runs the remote query while it holds the query's admission slot, and the handshake used to wait without one: a remote frontend that accepts the connection but never answers would hold the slot, and the connection thread, for good - neither KILL nor the query's timeout interrupts the wait. A dispatch's remote work is now bounded by query_retry_count x (connect + handshake + GetFlightInfo + closing a failed attempt): about 3 x 70 s with the catalog defaults.
   • Removed: isExplainStatement(); the scan's session fields and its stop(), coordinatorMustOutliveDispatch() and cannotBeRedispatched() overrides; the StatementContext fallback (stopScanNodeAtClose, handOverScanNodesToDeferredCoordinator, the step in close()); and the hand-over in the Arrow Flight deferral gate. The split assignment is what keeps a deferred Flight coordinator alive, as for any batch scan.
6. [fix](fe) A batch scan that plans with its first split (hive / iceberg / paimon / MaxCompute) starts its split generation while planning, and nothing stopped it when the plan was dropped: every successful CREATE JOB ... DO INSERT ... SELECT, every task of a streaming insert job whose SELECT reads such a table besides the TVF (before() plans the statement only to rewrite the TVF), an INSERT whose label is taken, a statement refused by a SQL block rule after planning, an http_stream load refused after planning, the plan an INSERT drops when its target table changes, a DELETE whose predicate reads another table (it falls back to DeleteFromUsingCommand, which plans again). The streaming generator of an iceberg scan then waits forever on a full backend queue, holding one of the 64 shared scheduleExecutor threads until the FE restarts; a streaming job did that every max_interval (10 s).
   • The statement owns what its planning started until the coordinator's exec() takes it over (SplitAssignment.start).
   • Where a statement drops a plan and goes on, it stops that plan's scans (ScanNode.stopAllUndispatched, which logs a failed split generation as one instead of throwing it as a failure to stop): EXPLAIN, the INSERT OVERWRITE probe, the INSERT planned again, DELETE, the plan a streaming task's before() builds to rewrite the TVF.
   • Where a statement ends, it stops what no coordinator took and forgets the rest (StatementContext.stopUndispatchedSplitAssignments): StatementContext.close(); the end of a binary COM_STMT_EXECUTE, whose context the prepared statement keeps until its next execution, so it must not keep the plan alive through an assignment; an http_stream request; every attempt of a streaming insert task (on the task thread, a canceled attempt too) and the attempt a cloud re-plan retries.
   • A stopped assignment drops the splits it still queues, so whatever still references it keeps none.
   • For a statement that ends without any of these, a generator still waiting on a full backend queue once the statement's timeout has passed stops the assignment itself, and logs the scan and the query that planned it. A generator that finished, or never filled a queue, holds no thread: its assignment goes with the plan.
7. [fix](fe) A backend's fetch created an unbounded split queue when it came before the generator's first split for that backend, so the generator never waited for that backend. Both now create the same bounded queue. Found while testing item 6.
8. [fix](regression-framework) The connections of the thread-local Doris accessors are closed once their thread ends (OpenedDorisConnections, swept from all three accessors, with strays kept until their thread ends and closed at the end of the run), and Awaitility no longer blames the suite that happens to be awaiting for another thread's uncaught exception.
9. [fix](regression-framework) A suite fails for an uncaught exception on a thread it started (UncaughtThreadFailures), including a thread that a task of Suite.thread() starts, which runs as the suite that submitted it; a suite that fails on something else carries those failures along as suppressed exceptions.
10. [fix](fe) Unrelated to remote Doris, found when an External run of this PR lost its FE: dropping or altering an ADBC catalog while a statement or a column-statistics load was still inside its driver crashed the FE (SIGSEGV in sqlite3FindTable). AdbcClient.close() closed the native AdbcDatabase at once, and the JNI driver first closes every connection still open on it, from the closing thread, under the threads using them. Now close() only refuses new calls, and the native driver is released by whoever leaves last: close() itself when no call is in flight, otherwise the last call to leave, which logs a failed release instead of failing its own work. close() still does not wait, so DROP / ALTER CATALOG are not held up. Self-contained: it can be picked without the rest of this PR.

No new outbound request: the existing Flight SQL request to the catalog's fe_arrow_hosts now happens only when the local query runs, and fewer times than before.

What changes besides:

• Every candidate backend runs a scan instance of a remote Doris scan, as for any batch scan. Most of them read nothing when the remote query returns few endpoints. A backend that holds an endpoint fetches its splits from the frontend twice, the second fetch learning at once that there are no more; any other backend fetches once. Neither fetch waits.
• The split count a SQL block rule sees for the scan is 0 instead of the number of endpoints.
• The remote query's time moves from the plan time to the schedule time of the profile.
• A remote failure fails the dispatch, with the same Failed to execute query message.
• Batch-mode hive / iceberg / paimon / MaxCompute scans plan and generate their splits as before. What changes is who stops that generation (item 6): besides the coordinator, the statement where it drops or ends a plan no coordinator took, or failing both the generator itself, if it is still waiting on a full backend queue at the statement's timeout. A stopped scan no longer keeps the splits it had queued. A retry with the same plan is refused once their split sources are released (item 2), and a backend's last fetch no longer waits 100 ms (item 3).
• A backend checks a cached connection to a frontend, or to another backend, before it uses it, with one non-blocking peek, and connects anew when the server has closed it (item 4).

Results

Remote queries each statement ran on the remote FE, counted from the remote FE's audit log, plus the catalog user's Flight SQL sessions left open after each statement. Local cluster, one FE and one BE, with the catalog pointing back at itself. "Before" is the previous head of this PR, which behaves like master on these statements.

Statement	Remote queries, before	Remote queries, after	Sessions left
SELECT ... FROM remote	1	1	0 / 0
EXPLAIN SELECT ...	0	0	0 / 0
/* a comment first */ EXPLAIN SELECT ...	1	0	0 / 0
INSERT OVERWRITE TABLE t SELECT ... FROM remote	2	1	0 / 0
INSERT ... WITH LABEL l SELECT ... (label free)	1	1	0 / 0
same again (label taken, refused)	1	0	0 / 0
CREATE JOB ... STARTS '2099-01-01' DO INSERT ... SELECT FROM remote	1	0	0 / 0
one-shot CREATE JOB ... AT CURRENT_TIMESTAMP (creation + its task)	1 (creation); task FAILED: NullPointerException ... "originStmt"	1 (the task); task SUCCESS	0 / 0

An Arrow Flight SQL client reading a remote Doris table: 3 rows returned, the remote session open (1) while the deferred query waits, and 0 after the client's next request finalizes it.

Classes, and how they call each other

• SplitAssignment: registerSource (planning), start (the coordinator takes it over; registers sources, then init -> SplitGenerator.startSplit), startWhilePlanning (a scan that plans with its first split), addCloseable, stop (unregisters, drops the queued splits, closes, rethrows a generation failure last), stopIfNotDispatched (the statement's end: the same unless a coordinator took it over, logging a generation failure instead of throwing it).
• ScanNode: start / stop delegate to the split assignment; startAll / stopAll for the coordinators; stopUndispatched / stopAllUndispatched (SplitAssignment.stopIfNotDispatched, no rethrow) for a plan the statement drops; coordinatorMustOutliveDispatch (unchanged: splitAssignment != null); cannotBeRedispatched (splitAssignment != null && splitAssignment.isStop()).
• FileQueryScanNode.createScanRangeLocations: batch mode creates the assignment and one split source per backend; it starts the assignment only when needsSampleSplit(), registering it with the statement first.
• Coordinator.exec / NereidsCoordinator.exec: ScanNode.startAll after admission, before the fragments are built; close / cancel: ScanNode.stopAll.
• RemoteDorisScanNode: convertPredicate builds the query; isBatchMode true, needsSampleSplit false, numApproximateSplits 0; startSplit -> executeQuery -> executeFlightSqlQuery (RemoteDorisFlightSession.open + execute, splitAssignment.addCloseable(session)) -> addToQueue(endpoints) -> finishSchedule.
• RemoteDorisFlightSession: open bounds the handshake by query_timeout_sec (it had no deadline); CloseSession bounded to 5s on close(), as before.
• StatementContext: no longer knows about scan nodes; it keeps the split assignments its planning started, and stopUndispatchedSplitAssignments stops those no coordinator took over and forgets all of them. Called by close(), MysqlConnectProcessor.handleExecute (end of a binary COM_STMT_EXECUTE), FrontendServiceImpl.initHttpStreamPlan, StmtExecutor.queryRetry (before a re-plan) and StreamingInsertTask.endAttempt (from AbstractStreamingTask.execute's finally). StmtExecutor.planCannotBeRedispatched reads the general rule.
• ScanNode.stopAllUndispatched on a plan dropped mid-statement: ExplainCommand, InsertOverwriteTableCommand (probe), InsertIntoTableCommand.initPlan (re-plan), DeleteFromCommand.run, StreamingInsertTask.before.
• ClientCacheHelper::get_client (BE, item 4): a cached client whose server closed the connection (ThriftClientImpl::peer_closed) is closed, and the next cached one tried or a new one opened; reopen_client shares the closing (_close_client).
• AdbcClient (item 10): withConnection -> enter (counts the call in, opens the database on first use) ... leave (counts it out; the last one out of a closed client calls release); close refuses new calls and calls release only when no call is in flight.
• Untouched: the BE's batch split-source read path and remote Doris reader, which already exist; the remote FE.

planning (translate / finalize)              Coordinator.exec (after admission)                 close / cancel
RemoteDorisScanNode.convertPredicate         ScanNode.startAll
  '- builds the remote query (EXPLAIN)         '- SplitAssignment.start
FileQueryScanNode.createScanRangeLocations          |- registers the split sources
  |- new SplitAssignment (not started)              '- RemoteDorisScanNode.startSplit
  '- one SplitSource per BE (an id only),                |- RemoteDorisFlightSession.open ---- handshake -----> remote FE
     scan range = TSplitSource{id}                       |- session.execute ---------------- GetFlightInfo --> remote query
                                                         |- splitAssignment.addCloseable(session)
                                                         '- addToQueue(endpoints), finishSchedule
                                             fragments -> local BE: fetchSplitBatch -> DoGet(ticket) -> remote BE
                                                                                                    ScanNode.stopAll
                                                                                                      '- SplitAssignment.stop
                                                                                                           |- unregisters sources
                                                                                                           '- session.close -- CloseSession --> remote FE

Release note

A remote Doris catalog (use_arrow_flight = true) runs the query on the remote cluster only when the local query runs. EXPLAIN, statements that fail before they run, and the plans INSERT OVERWRITE and CREATE JOB only inspect no longer reach the remote cluster, and INSERT OVERWRITE runs the remote query once instead of twice. Jobs and streaming insert jobs can read remote Doris tables; before, they failed with a NullPointerException. A query with a batch-mode external table scan that fails with an RPC error is no longer retried with the same plan, which could only fail with "Split source ... is released"; it reports the original error. A CREATE JOB, a streaming insert job, or a statement refused or failing after planning, whose plan reads a large iceberg table in batch mode no longer leaves a frontend thread generating splits forever; enough of them stalled every batch-mode external table scan on that frontend. The frontend no longer keeps most of the splits of a huge batch-mode scan in memory when a backend fetches before it was assigned a split. A query that reads remote Doris, or a batch-mode external table, no longer fails with "Failed to get batch of split source" right after the frontend running it restarts. A remote Doris query whose remote frontend accepts the connection but never answers gives up after the catalog's query_timeout_sec instead of waiting for good. Fix an FE crash (SIGSEGV in the ADBC driver) when an ADBC catalog is dropped or altered while a statement or a statistics load is still reading it.

Check List (For Author)

• Test

  • Regression test
  • Unit Test
  • Manual test (add detailed scripts or steps below)
  • No need to test or manual test. Explain why:
    • This is a refactor/code format and no logic has been changed.
    • Previous test can cover this change.
    • No code files have been changed.
    • Other reason

  Unit tests, fe-core: SplitAssignmentTest, StatementContextTest, MysqlConnectProcessorExecuteEndTest, MysqlConnectProcessorCursorFetchTest, StreamingInsertTaskAttemptEndTest, StreamingInsertTaskAuditTest, DeleteFromCommandTest, FrontendServiceImplHttpStreamPlanTest, StmtExecutorReplanRetryTest, ScanNodeDispatchTest, RemoteDorisScanNodeTest, ConnectorStatementScopeTest, ArrowFlightDeferralGateTest, OldCoordinatorTest, NereidsCoordinatorTest, ExecuteCommandTest, StmtExecutorTest, ProtocolCapabilityWiringTest and the PluginDrivenScanNode batch mode / scan profile / read txn / compatibility tests: 179, all passing; checkstyle 0. AdbcClientTest: 11, all passing with the real JNI bridge and SQLite driver (0 skipped). Regression framework: 58 (those two commits are unchanged since that run), including SuiteContextDorisConnectionsTest and SuiteThreadFailuresTest, which run the accessors and whole suites. BE: client_cache_test, 3 cases, all passing under ASAN; with the check in get_client taken out, the stale-connection case fails. Each fix was checked against its test: undoing it fails the test, for every fix but one - the plan an INSERT drops when its target table changes between planning and locking has no unit test. Regression, on a local cluster: the 13 suites of external_table_p0/remote_doris, including test_remote_doris_flight_session, which binds a SQL block rule that refuses every remote query of the catalog user and checks that EXPLAIN (also behind a comment), a refused INSERT and CREATE JOB never reach the remote FE, and that a job's insert task reads the remote table, its rows going through a generated .out; prepared_stmt_p0 (server-side prepared statements, through the new end of a binary COM_STMT_EXECUTE), load_p0/http_stream, and delete_p0/test_delete_where_in (a DELETE falling back to DeleteFromUsingCommand): 29 suites, all passing. Manual, on the same cluster: the per-statement counts and the Arrow Flight client check above; 20 remote Doris queries, then three FE restarts with 20 remote Doris queries right after each: none failed, and the backend closed 9 cached connections the restarted frontend had closed (item 4) - earlier local runs without item 4 logged remote Doris queries failing right after a frontend restart with Failed to get batch of split source; a remote Doris catalog whose fe_arrow_hosts accepts the connection and never answers (query_timeout_sec 3, query_retry_count 2): the query fails after 6 s with Failed to execute query instead of waiting for good (item 5). The batch-mode hive / iceberg / paimon suites were not run locally (no docker environment for them): their planning and split generation are unchanged, but who stops that generation is not (item 6), so those suites are the ones to watch in CI.

• Behavior changed:

  • No.
  • Yes. See the release note, and "What changes besides".

• Does this need documentation?

  • No.
  • Yes.

Check List (For Reviewer who merge this PR)

• Confirm the release note
• Confirm test cases
• Confirm document
• Add branch pick label

[hello-stephen]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?

[morningman]

run buildall

[morningman]

run buildall

[hello-stephen]

TPC-H: Total hot run time: 27542 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit aefca1782c94df7961e830e4348658204176d5c5, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17729	3866	3868	3866
q2	2148	372	302	302
q3	10134	1417	805	805
q4	4688	485	350	350
q5	7479	825	556	556
q6	181	167	137	137
q7	747	784	587	587
q8	9333	1428	1571	1428
q9	5572	4186	4169	4169
q10	6827	1324	1035	1035
q11	437	275	252	252
q12	636	413	299	299
q13	18047	2608	1987	1987
q14	268	261	248	248
q15	q16	735	724	670	670
q17	1805	1091	1006	1006
q18	6556	5618	5546	5546
q19	1179	1169	993	993
q20	483	402	261	261
q21	5151	2908	2746	2746
q22	418	356	299	299
Total cold run time: 100553 ms
Total hot run time: 27542 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4161	4068	4073	4068
q2	708	580	527	527
q3	4475	4854	4364	4364
q4	2227	2328	1462	1462
q5	4197	4124	4113	4113
q6	228	174	128	128
q7	1741	1627	1404	1404
q8	2198	2304	2212	2212
q9	7553	7670	7525	7525
q10	3746	3782	3161	3161
q11	578	395	369	369
q12	777	741	524	524
q13	2410	2859	2143	2143
q14	302	310	279	279
q15	q16	706	717	635	635
q17	7718	7261	7137	7137
q18	12182	11126	11827	11126
q19	1185	1043	1077	1043
q20	2255	2214	1938	1938
q21	5412	4343	4552	4343
q22	541	480	400	400
Total cold run time: 65300 ms
Total hot run time: 58901 ms

[hello-stephen]

TPC-DS: Total hot run time: 152352 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit aefca1782c94df7961e830e4348658204176d5c5, data reload: false

query5	4336	602	479	479
query6	442	216	202	202
query7	4808	512	304	304
query8	319	182	165	165
query9	8815	3985	3972	3972
query10	442	308	252	252
query11	5804	3586	3262	3262
query12	155	93	89	89
query13	1260	604	420	420
query14	6524	4487	4196	4196
query14_1	4016	3950	3916	3916
query15	199	192	177	177
query16	955	418	405	405
query17	872	656	516	516
query18	2413	462	320	320
query19	192	173	140	140
query20	82	79	80	79
query21	218	133	114	114
query22	13002	12902	12862	12862
query23	13945	13021	12486	12486
query23_1	12361	12561	12445	12445
query24	7286	1178	666	666
query24_1	663	687	685	685
query25	528	417	345	345
query26	1247	306	155	155
query27	2735	541	320	320
query28	4541	1949	2006	1949
query29	1575	699	503	503
query30	301	217	184	184
query31	902	745	627	627
query32	165	98	85	85
query33	511	302	234	234
query34	1183	1119	626	626
query35	718	739	624	624
query36	789	774	725	725
query37	140	100	86	86
query38	1810	1765	1709	1709
query39	693	703	675	675
query39_1	649	660	667	660
query40	220	119	100	100
query41	66	64	62	62
query42	92	92	92	92
query43	333	342	296	296
query44	1344	703	696	696
query45	182	179	164	164
query46	1065	1159	719	719
query47	1483	1478	1400	1400
query48	373	411	287	287
query49	591	397	300	300
query50	946	338	263	263
query51	10592	10507	10337	10337
query52	87	86	75	75
query53	243	257	174	174
query54	241	196	189	189
query55	79	72	72	72
query56	222	232	215	215
query57	1408	1460	1365	1365
query58	285	260	253	253
query59	2003	2086	1863	1863
query60	274	243	228	228
query61	149	148	138	138
query62	394	321	263	263
query63	215	172	178	172
query64	2762	970	790	790
query65	3455	3410	3388	3388
query66	1830	428	298	298
query67	20509	20117	19828	19828
query68	3318	1584	851	851
query69	423	301	261	261
query70	903	817	832	817
query71	287	245	218	218
query72	2848	2667	2381	2381
query73	807	784	425	425
query74	4636	4473	4291	4291
query75	2331	2294	1935	1935
query76	2430	1111	721	721
query77	369	408	311	311
query78	9182	9093	8521	8521
query79	1183	1216	760	760
query80	542	475	390	390
query81	539	318	279	279
query82	312	160	128	128
query83	315	225	209	209
query84	322	151	123	123
query85	933	525	377	377
query86	328	248	215	215
query87	2005	1981	1829	1829
query88	3631	2764	2759	2759
query89	352	279	248	248
query90	1800	187	174	174
query91	173	157	127	127
query92	101	82	79	79
query93	1339	1375	838	838
query94	529	331	292	292
query95	663	374	444	374
query96	1120	777	328	328
query97	2412	2427	2305	2305
query98	158	150	152	150
query99	714	726	617	617
Total cold run time: 235598 ms
Total hot run time: 152352 ms

[hello-stephen]

ClickBench: Total hot run time: 24.15 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit aefca1782c94df7961e830e4348658204176d5c5, data reload: false

query1	0.00	0.00	0.01
query2	0.09	0.05	0.05
query3	0.26	0.14	0.13
query4	1.61	0.14	0.14
query5	0.25	0.22	0.22
query6	1.15	0.99	0.95
query7	0.04	0.01	0.01
query8	0.06	0.04	0.04
query9	0.39	0.34	0.34
query10	0.55	0.59	0.58
query11	0.21	0.16	0.14
query12	0.18	0.15	0.15
query13	0.48	0.47	0.48
query14	0.96	0.97	0.96
query15	0.62	0.56	0.61
query16	0.32	0.31	0.32
query17	1.06	1.06	1.13
query18	0.22	0.21	0.20
query19	2.03	1.96	2.00
query20	0.02	0.02	0.01
query21	15.49	0.20	0.14
query22	4.88	0.05	0.05
query23	16.14	0.29	0.12
query24	2.96	0.43	0.33
query25	0.10	0.05	0.04
query26	0.74	0.22	0.14
query27	0.05	0.05	0.03
query28	3.61	0.78	0.34
query29	12.54	4.01	3.21
query30	0.28	0.15	0.15
query31	2.78	0.56	0.32
query32	3.23	0.59	0.49
query33	3.14	3.26	3.21
query34	15.66	3.93	3.30
query35	3.23	3.20	3.25
query36	0.55	0.42	0.42
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.03	0.03
query40	0.17	0.15	0.16
query41	0.09	0.03	0.03
query42	0.03	0.04	0.04
query43	0.04	0.04	0.03
Total cold run time: 96.39 s
Total hot run time: 24.15 s

[morningman]

run buildall

[hello-stephen]

TPC-H: Total hot run time: 27350 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 82309eb1ae55c1f055d88c477586ed96827fd88f, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17587	3745	3771	3745
q2	2181	380	297	297
q3	10094	1409	783	783
q4	4687	475	349	349
q5	7482	834	560	560
q6	181	170	134	134
q7	750	782	616	616
q8	9311	1395	1495	1395
q9	5516	4174	4178	4174
q10	6829	1339	1026	1026
q11	433	268	251	251
q12	636	423	295	295
q13	18101	2611	1986	1986
q14	276	265	234	234
q15	q16	738	724	662	662
q17	1773	1132	1005	1005
q18	6465	5669	5536	5536
q19	1310	1187	1045	1045
q20	471	397	268	268
q21	5785	2895	2696	2696
q22	435	348	293	293
Total cold run time: 101041 ms
Total hot run time: 27350 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4132	4017	4000	4000
q2	710	563	513	513
q3	4444	4833	4292	4292
q4	2217	2293	1430	1430
q5	4227	4093	4091	4091
q6	228	175	125	125
q7	1713	1631	1410	1410
q8	2158	2116	2268	2116
q9	7613	7553	7581	7553
q10	3759	3935	3157	3157
q11	540	409	370	370
q12	753	726	534	534
q13	2425	2835	2150	2150
q14	299	304	265	265
q15	q16	687	716	641	641
q17	7726	7145	7010	7010
q18	11862	11102	11859	11102
q19	1270	1056	1052	1052
q20	2243	2218	1963	1963
q21	5257	4398	4500	4398
q22	524	470	410	410
Total cold run time: 64787 ms
Total hot run time: 58582 ms

[hello-stephen]

TPC-DS: Total hot run time: 152869 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 82309eb1ae55c1f055d88c477586ed96827fd88f, data reload: false

query5	4328	604	460	460
query6	422	224	198	198
query7	4809	540	297	297
query8	321	177	162	162
query9	8819	3956	3961	3956
query10	443	313	263	263
query11	5831	3567	3245	3245
query12	143	88	87	87
query13	1244	603	448	448
query14	6543	4520	4261	4261
query14_1	3955	3939	3921	3921
query15	203	194	185	185
query16	982	457	421	421
query17	904	668	534	534
query18	2449	464	336	336
query19	197	180	150	150
query20	84	81	80	80
query21	225	130	112	112
query22	13036	13033	14059	13033
query23	14604	13538	12862	12862
query23_1	12863	12606	12512	12512
query24	7260	1109	671	671
query24_1	697	667	686	667
query25	557	447	369	369
query26	1249	300	168	168
query27	2761	555	332	332
query28	4555	1972	1943	1943
query29	1673	676	490	490
query30	308	217	171	171
query31	896	747	629	629
query32	150	86	90	86
query33	490	308	237	237
query34	1194	1062	604	604
query35	710	736	625	625
query36	789	788	675	675
query37	151	99	94	94
query38	1821	1758	1673	1673
query39	707	673	686	673
query39_1	651	644	644	644
query40	214	118	96	96
query41	63	60	59	59
query42	93	88	89	88
query43	327	335	294	294
query44	1329	702	698	698
query45	176	164	160	160
query46	1045	1154	722	722
query47	1480	1482	1400	1400
query48	404	406	297	297
query49	570	405	291	291
query50	912	335	246	246
query51	10676	10635	10301	10301
query52	87	85	73	73
query53	234	248	177	177
query54	264	232	178	178
query55	75	75	68	68
query56	237	213	213	213
query57	1358	1378	1391	1378
query58	319	258	249	249
query59	1965	2055	1826	1826
query60	285	242	224	224
query61	146	146	145	145
query62	397	321	260	260
query63	213	179	181	179
query64	2824	1014	819	819
query65	3466	3395	3385	3385
query66	1808	410	299	299
query67	20058	20215	20043	20043
query68	3006	1501	953	953
query69	409	301	250	250
query70	916	802	798	798
query71	299	229	204	204
query72	2701	2530	2270	2270
query73	856	767	424	424
query74	4629	4489	4274	4274
query75	2295	2267	1920	1920
query76	2291	1087	718	718
query77	355	382	290	290
query78	9101	9023	8488	8488
query79	1320	1131	731	731
query80	913	450	357	357
query81	581	320	270	270
query82	867	153	125	125
query83	296	222	195	195
query84	310	147	113	113
query85	900	500	378	378
query86	386	245	233	233
query87	1986	1977	1812	1812
query88	3611	2725	2697	2697
query89	351	287	245	245
query90	1855	181	178	178
query91	169	155	130	130
query92	106	89	89	89
query93	1443	1407	856	856
query94	612	343	313	313
query95	650	370	417	370
query96	1039	775	342	342
query97	2466	2437	2331	2331
query98	160	155	141	141
query99	734	711	612	612
Total cold run time: 237301 ms
Total hot run time: 152869 ms

[hello-stephen]

ClickBench: Total hot run time: 24.19 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 82309eb1ae55c1f055d88c477586ed96827fd88f, data reload: false

query1	0.01	0.01	0.01
query2	0.09	0.05	0.05
query3	0.27	0.14	0.14
query4	1.61	0.14	0.14
query5	0.25	0.23	0.23
query6	1.16	0.94	0.94
query7	0.04	0.01	0.00
query8	0.05	0.04	0.04
query9	0.41	0.36	0.36
query10	0.58	0.60	0.58
query11	0.20	0.16	0.15
query12	0.18	0.15	0.15
query13	0.46	0.48	0.49
query14	0.96	0.97	0.95
query15	0.62	0.59	0.60
query16	0.31	0.32	0.33
query17	1.12	1.11	1.12
query18	0.22	0.20	0.21
query19	1.99	1.94	1.89
query20	0.02	0.01	0.02
query21	15.49	0.20	0.14
query22	4.99	0.05	0.05
query23	16.12	0.31	0.12
query24	3.00	0.41	0.32
query25	0.12	0.05	0.04
query26	0.72	0.20	0.14
query27	0.04	0.03	0.04
query28	3.53	0.83	0.34
query29	12.49	4.08	3.20
query30	0.29	0.15	0.16
query31	2.77	0.57	0.31
query32	3.23	0.59	0.48
query33	3.13	3.23	3.23
query34	15.55	3.97	3.28
query35	3.21	3.26	3.23
query36	0.56	0.44	0.43
query37	0.09	0.07	0.06
query38	0.06	0.04	0.04
query39	0.03	0.03	0.03
query40	0.17	0.14	0.14
query41	0.08	0.03	0.03
query42	0.04	0.03	0.03
query43	0.05	0.04	0.03
Total cold run time: 96.31 s
Total hot run time: 24.19 s

[morningman]

/review

[github-actions]

Request changes: two P1 correctness gaps remain.

Findings

1. Streaming-task cancellation does not give the new statement cleanup a post-quiescence owner. STOPPED skips it, and PAUSED can run it before planning finishes, so the pre-rewrite remote Doris Flight SQL session can still leak until remote wait_timeout.
2. The process-wide Awaitility setting removes uncaught raw-worker failures from every suite verdict before the framework has owner-local propagation, allowing product failures to turn green.

Critical checkpoint conclusions

• Goal and proof: The PR closes remote Doris sessions on several previously missed statement paths, but does not accomplish that goal for canceled streaming attempts. The direct cleanup test does not exercise STOPPED or cancellation racing planning.
• Scope and clarity: The FE cleanup changes are focused; the Awaitility switch is process-wide and therefore much broader than the connection-ownership fix.
• Concurrency: The streaming scheduler task thread races the PAUSE/STOP control thread over ctx and stmtExecutor; the job lock does not quiesce before(). Regression suites also use raw worker threads; join() provides ordering but no throwable propagation. Connection-map add/drain/remove synchronization otherwise survived the interleaving review, with heavy closes outside its monitor. No new lock-order or deadlock defect was found.
• Lifecycle: Prepared execution closes the correct statement generation, Arrow-deferred scans are handed to the coordinator, AutoClose owners finish synchronous use before teardown, and legacy/Nereids coordinators now attempt every scan-node stop. The remaining broken lifecycle is the canceled streaming attempt described inline. No static-initialization issue applies.
• Configuration and compatibility: No configuration, wire/storage format, symbol, rolling-upgrade, persistence, transaction, data-write, or FE-BE variable-passing change is introduced.
• Parallel and conditional paths: Direct/forwarded prepared execution, normal/Arrow ownership, AutoClose callers, legacy/Nereids coordinators, success/failure/retry, PAUSED/STOPPED, and regression suite/STRAY connection ownership were checked. Per-node cleanup isolation and connection-transfer conditions are sound; the canceled-task cleanup suppression is not sound with late planning.
• Tests and results: Added tests cover positive direct cleanup, coordinator failure isolation, prepared execution, and connection tracking, but omit the two cancellation paths and owner-local raw-worker failure propagation. This was a static-only review: the review prompt prohibited builds/tests, so author/CI claims were not treated as independent validation and no local test result is claimed.
• Observability: Existing logs are adequate to diagnose cleanup failures, but logging cannot replace deterministic session close or a failing suite verdict.
• Performance: No substantiated CPU, memory, or complexity regression was found in the per-node stop or connection tracking changes.
• User focus: No additional focus was supplied; the entire 13-file change set and relevant upstream/downstream call chains were reviewed.

All other candidates were either duplicates or dismissed with concrete code evidence. Overall state: changes requested pending fixes for both P1 findings and negative coverage for their failure paths.

[github-actions]

[P1] Close canceled attempts only after planning has quiesced

This cleanup is not guaranteed to run at a terminal task boundary. STOPPED only calls cancelAllTasks(false), which marks the task CANCELED; StreamingInsertJob.updateJobStatus force-cleans only PAUSED, while AbstractStreamingTask.execute() skips its own cleanup for every canceled task, so this line is never reached. There is also a PAUSED race before stmtExecutor is assigned: the control thread can close and clear the statement while baseCommand.initPlan(...) is still running, after which a RemoteDorisScanNode can register into the already-drained StatementContext and the canceled task again skips a final close. In both cases the pre-rewrite plan has no coordinator to close that Flight SQL session, leaving it until remote wait_timeout. Please make the task thread perform one final close after before()/run() can no longer register nodes for every terminal status, and cover STOPPED plus cancel-during-planning.

[github-actions]

[P1] Preserve a suite verdict for raw worker failures

This avoids cross-suite misattribution by making uncaught raw-worker failures unable to fail any suite. The tree still has many Thread.start/new Thread workers that run SQL or assertions and whose parents only call join(); join() never rethrows, and the framework installs no per-suite uncaught-error collector. Suite.thread() can propagate through its future, but those existing raw-thread callers do not use it. The manual result in this PR also explicitly shows the new behavior: a failing daemon only prints to stderr and fails no suite. Please add owner-local propagation (for example, migrate these workers to futures plus get(), or add a per-suite collector checked after joins) before disabling the only process-level signal, so product regressions cannot turn green.

[morningman]

run buildall

[hello-stephen]

TPC-H: Total hot run time: 27814 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c7d971216e1f65b4133130fc4fddc0c47246e836, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17731	3973	3887	3887
q2	2184	356	311	311
q3	10066	1380	797	797
q4	4688	472	350	350
q5	7489	826	534	534
q6	179	173	138	138
q7	765	768	595	595
q8	9302	1523	1606	1523
q9	5412	4194	4162	4162
q10	6813	1310	1034	1034
q11	428	266	244	244
q12	634	409	300	300
q13	18050	2603	1983	1983
q14	258	254	237	237
q15	q16	729	714	652	652
q17	1786	1121	995	995
q18	6458	5563	5508	5508
q19	1214	1216	981	981
q20	463	393	268	268
q21	5411	3060	3015	3015
q22	431	347	300	300
Total cold run time: 100491 ms
Total hot run time: 27814 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4558	4479	4466	4466
q2	728	561	531	531
q3	4725	5278	4609	4609
q4	2192	2365	1447	1447
q5	4659	4429	4437	4429
q6	232	172	135	135
q7	1853	1730	1466	1466
q8	2294	2031	2056	2031
q9	7293	6834	6822	6822
q10	3611	3537	3091	3091
q11	518	368	345	345
q12	735	702	514	514
q13	2293	2586	1997	1997
q14	266	269	260	260
q15	q16	660	678	603	603
q17	7261	6716	6617	6617
q18	11770	10966	11778	10966
q19	1112	993	1014	993
q20	2207	2179	1910	1910
q21	4933	4070	4291	4070
q22	509	448	403	403
Total cold run time: 64409 ms
Total hot run time: 57705 ms

[hello-stephen]

TPC-DS: Total hot run time: 151973 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c7d971216e1f65b4133130fc4fddc0c47246e836, data reload: false

query5	4310	573	453	453
query6	432	204	189	189
query7	4863	555	305	305
query8	316	178	168	168
query9	8806	3967	3954	3954
query10	453	310	261	261
query11	5810	3540	3230	3230
query12	147	90	88	88
query13	1251	615	403	403
query14	6502	4522	4219	4219
query14_1	3961	3980	3922	3922
query15	203	199	175	175
query16	965	450	434	434
query17	907	672	537	537
query18	2416	460	325	325
query19	198	177	147	147
query20	84	84	80	80
query21	216	134	112	112
query22	13003	12917	12860	12860
query23	13831	12943	12427	12427
query23_1	12454	12400	12471	12400
query24	7382	1174	669	669
query24_1	662	687	690	687
query25	552	431	365	365
query26	1259	298	173	173
query27	2690	589	335	335
query28	4587	2020	1957	1957
query29	1626	698	488	488
query30	296	218	177	177
query31	881	759	632	632
query32	152	85	90	85
query33	505	291	237	237
query34	1228	1093	653	653
query35	716	774	624	624
query36	801	782	724	724
query37	144	103	92	92
query38	1821	1744	1703	1703
query39	676	705	678	678
query39_1	648	634	650	634
query40	214	119	102	102
query41	64	63	62	62
query42	95	87	89	87
query43	330	340	291	291
query44	1377	724	701	701
query45	194	174	161	161
query46	1067	1172	736	736
query47	1489	1493	1411	1411
query48	406	432	280	280
query49	592	394	292	292
query50	948	358	257	257
query51	10552	10443	10096	10096
query52	86	92	75	75
query53	241	246	181	181
query54	257	201	184	184
query55	78	73	72	72
query56	246	207	208	207
query57	1566	1439	1321	1321
query58	283	254	261	254
query59	1989	2049	1856	1856
query60	272	247	221	221
query61	154	145	143	143
query62	398	316	265	265
query63	206	175	178	175
query64	2801	956	785	785
query65	3469	3415	3410	3410
query66	1784	412	297	297
query67	20268	19932	19867	19867
query68	3094	1538	943	943
query69	392	300	259	259
query70	886	837	820	820
query71	304	231	211	211
query72	2570	2505	2270	2270
query73	857	805	450	450
query74	4615	4484	4272	4272
query75	2305	2298	1935	1935
query76	2394	1135	778	778
query77	371	415	313	313
query78	9248	9223	8498	8498
query79	1266	1236	772	772
query80	590	483	404	404
query81	540	327	278	278
query82	667	163	133	133
query83	316	223	197	197
query84	323	156	120	120
query85	958	460	390	390
query86	333	233	232	232
query87	1971	1964	1840	1840
query88	3608	2717	2699	2699
query89	340	284	243	243
query90	1895	178	174	174
query91	168	157	130	130
query92	105	87	105	87
query93	1500	1528	853	853
query94	530	333	305	305
query95	657	444	332	332
query96	1051	740	306	306
query97	2414	2446	2331	2331
query98	168	145	142	142
query99	718	737	607	607
Total cold run time: 235892 ms
Total hot run time: 151973 ms

[hello-stephen]

ClickBench: Total hot run time: 24.1 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c7d971216e1f65b4133130fc4fddc0c47246e836, data reload: false

query1	0.01	0.01	0.01
query2	0.10	0.05	0.05
query3	0.26	0.14	0.13
query4	1.61	0.14	0.14
query5	0.25	0.23	0.22
query6	1.15	0.93	0.91
query7	0.04	0.01	0.00
query8	0.05	0.04	0.04
query9	0.38	0.34	0.34
query10	0.57	0.58	0.54
query11	0.20	0.15	0.14
query12	0.18	0.14	0.14
query13	0.46	0.47	0.45
query14	0.96	0.95	0.93
query15	0.60	0.59	0.58
query16	0.32	0.35	0.32
query17	1.10	1.11	1.10
query18	0.22	0.20	0.19
query19	2.02	1.98	1.91
query20	0.02	0.01	0.01
query21	15.44	0.23	0.14
query22	4.72	0.06	0.05
query23	16.14	0.31	0.12
query24	2.97	0.42	0.33
query25	0.11	0.05	0.05
query26	0.74	0.20	0.16
query27	0.05	0.04	0.04
query28	3.56	0.73	0.35
query29	12.54	4.10	3.32
query30	0.28	0.17	0.15
query31	2.77	0.55	0.31
query32	3.22	0.59	0.48
query33	3.14	3.20	3.27
query34	15.76	3.92	3.26
query35	3.25	3.21	3.24
query36	0.56	0.45	0.42
query37	0.09	0.06	0.06
query38	0.05	0.03	0.04
query39	0.03	0.03	0.03
query40	0.17	0.15	0.14
query41	0.09	0.03	0.03
query42	0.04	0.04	0.04
query43	0.04	0.04	0.03
Total cold run time: 96.26 s
Total hot run time: 24.1 s

[hello-stephen]

FE Regression Coverage Report

Increment line coverage 86.36% (19/22) 🎉
Increment coverage report
Complete coverage report

[hello-stephen]

FE UT Coverage Report

Increment line coverage 90.91% (20/22) 🎉
Increment coverage report
Complete coverage report

[morningman]

run buildall

dismiss

[hello-stephen]

TPC-H: Total hot run time: 28712 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7b33695a994d4f04c1ecc34a2bd309409db1b7a9, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17586	5565	5531	5531
q2	2122	307	264	264
q3	10239	1408	834	834
q4	4672	469	344	344
q5	7440	787	525	525
q6	184	187	158	158
q7	773	797	601	601
q8	9229	1323	1056	1056
q9	5759	4454	4449	4449
q10	6809	1295	1041	1041
q11	418	257	238	238
q12	634	415	288	288
q13	18044	3077	2375	2375
q14	268	267	250	250
q15	q16	743	725	673	673
q17	1165	737	566	566
q18	7203	6229	6220	6220
q19	1271	960	603	603
q20	395	334	232	232
q21	5431	2199	2365	2199
q22	371	309	265	265
Total cold run time: 100756 ms
Total hot run time: 28712 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	6152	6097	6083	6083
q2	681	561	590	561
q3	5199	5397	4942	4942
q4	2000	2117	1526	1526
q5	5248	5205	5995	5205
q6	277	195	150	150
q7	2330	2058	1770	1770
q8	3147	2773	2791	2773
q9	8296	8217	8362	8217
q10	3831	3724	3292	3292
q11	672	440	412	412
q12	697	723	555	555
q13	2941	3258	2513	2513
q14	305	333	297	297
q15	q16	722	747	665	665
q17	8457	7604	7406	7406
q18	13099	12246	13066	12246
q19	907	832	811	811
q20	2157	2159	1950	1950
q21	5711	4669	4750	4669
q22	507	502	410	410
Total cold run time: 73336 ms
Total hot run time: 66453 ms

[hello-stephen]

FE Regression Coverage Report

Increment line coverage 18.56% (18/97) 🎉
Increment coverage report
Complete coverage report

[morningman]

run buildall

[hello-stephen]

TPC-H: Total hot run time: 29284 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a9535b5d3236d39715f7b549a492cfc4bfa1d0ce, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17597	5702	5549	5549
q2	2118	306	261	261
q3	10238	1415	846	846
q4	4678	482	357	357
q5	7443	804	531	531
q6	180	189	159	159
q7	793	807	618	618
q8	9239	1329	1138	1138
q9	5779	4444	4461	4444
q10	6822	1289	1031	1031
q11	458	266	241	241
q12	645	411	293	293
q13	18038	3073	2403	2403
q14	277	257	244	244
q15	q16	743	727	673	673
q17	1192	737	578	578
q18	7052	6279	6235	6235
q19	1110	945	605	605
q20	399	343	225	225
q21	5282	2548	2629	2548
q22	398	356	305	305
Total cold run time: 100481 ms
Total hot run time: 29284 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	6606	6625	6764	6625
q2	768	612	555	555
q3	5559	5633	5245	5245
q4	2091	2183	1599	1599
q5	5800	5648	5801	5648
q6	242	192	154	154
q7	2272	2071	1837	1837
q8	3175	2785	2752	2752
q9	8204	8167	7939	7939
q10	3653	3617	3238	3238
q11	587	398	368	368
q12	664	714	497	497
q13	2764	3059	2394	2394
q14	283	285	252	252
q15	q16	664	689	617	617
q17	7743	6998	6845	6845
q18	13053	12337	13068	12337
q19	936	796	805	796
q20	2218	2181	1934	1934
q21	5975	4776	4962	4776
q22	510	446	438	438
Total cold run time: 73767 ms
Total hot run time: 66846 ms

[hello-stephen]

TPC-DS: Total hot run time: 152201 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a9535b5d3236d39715f7b549a492cfc4bfa1d0ce, data reload: false

query5	4307	580	472	472
query6	425	217	192	192
query7	4797	442	226	226
query8	318	184	172	172
query9	8758	3980	4015	3980
query10	469	307	245	245
query11	6042	3458	3193	3193
query12	143	84	81	81
query13	1242	432	313	313
query14	6528	4873	4529	4529
query14_1	4282	4321	4283	4283
query15	208	199	186	186
query16	926	441	423	423
query17	905	680	571	571
query18	2421	428	308	308
query19	199	174	135	135
query20	82	80	81	80
query21	212	137	116	116
query22	13076	13071	12812	12812
query23	13332	12623	12178	12178
query23_1	12393	12362	12193	12193
query24	7103	854	464	464
query24_1	503	459	462	459
query25	508	391	355	355
query26	1228	250	129	129
query27	2786	439	268	268
query28	4605	1914	1915	1914
query29	1548	578	433	433
query30	297	217	176	176
query31	897	762	643	643
query32	145	86	90	86
query33	517	292	225	225
query34	904	851	501	501
query35	743	765	657	657
query36	799	807	714	714
query37	131	104	92	92
query38	1812	1764	1670	1670
query39	723	724	717	717
query39_1	676	674	678	674
query40	213	117	93	93
query41	68	63	62	62
query42	88	87	80	80
query43	372	390	324	324
query44	1336	694	689	689
query45	186	176	155	155
query46	848	970	567	567
query47	2967	3012	2818	2818
query48	292	312	218	218
query49	567	408	282	282
query50	683	289	209	209
query51	10500	10498	10444	10444
query52	79	87	75	75
query53	186	236	154	154
query54	228	192	175	175
query55	72	69	65	65
query56	222	217	210	210
query57	1619	1582	1542	1542
query58	273	256	248	248
query59	2370	2387	2146	2146
query60	300	232	223	223
query61	149	149	140	140
query62	388	337	288	288
query63	204	159	169	159
query64	2764	975	823	823
query65	3485	3403	3414	3403
query66	1782	417	294	294
query67	20175	20200	20130	20130
query68	2967	1026	607	607
query69	391	290	249	249
query70	922	857	833	833
query71	312	249	200	200
query72	2691	2718	2283	2283
query73	541	538	295	295
query74	4620	4487	4331	4331
query75	2520	2287	1935	1935
query76	2290	1033	630	630
query77	365	400	301	301
query78	9003	9053	8477	8477
query79	914	875	522	522
query80	500	419	357	357
query81	531	319	275	275
query82	211	135	112	112
query83	243	198	185	185
query84	303	116	105	105
query85	765	412	402	402
query86	255	234	234	234
query87	2002	1957	1873	1873
query88	3641	2691	2673	2673
query89	340	298	260	260
query90	2114	187	184	184
query91	156	147	124	124
query92	99	81	87	81
query93	1022	998	565	565
query94	457	311	246	246
query95	585	343	318	318
query96	695	555	246	246
query97	2443	2419	2290	2290
query98	162	149	150	149
query99	762	769	640	640
Total cold run time: 234030 ms
Total hot run time: 152201 ms

[hello-stephen]

ClickBench: Total hot run time: 24.49 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a9535b5d3236d39715f7b549a492cfc4bfa1d0ce, data reload: false

query1	0.01	0.01	0.00
query2	0.09	0.05	0.05
query3	0.27	0.12	0.12
query4	1.61	0.12	0.12
query5	0.24	0.26	0.25
query6	1.18	0.69	0.68
query7	0.04	0.01	0.01
query8	0.05	0.03	0.04
query9	0.43	0.34	0.34
query10	0.58	0.57	0.56
query11	0.21	0.14	0.14
query12	0.19	0.14	0.14
query13	0.48	0.49	0.50
query14	0.98	0.97	0.98
query15	0.64	0.60	0.60
query16	0.35	0.32	0.35
query17	1.18	1.15	1.13
query18	0.22	0.22	0.22
query19	2.15	2.05	1.96
query20	0.02	0.01	0.01
query21	15.49	0.26	0.14
query22	4.42	0.05	0.05
query23	16.27	0.32	0.11
query24	3.20	0.61	0.41
query25	0.11	0.08	0.05
query26	1.08	0.27	0.15
query27	0.04	0.04	0.03
query28	1.95	0.62	0.42
query29	12.51	4.16	3.32
query30	0.28	0.13	0.13
query31	2.78	0.59	0.34
query32	3.22	0.62	0.53
query33	3.15	3.24	3.20
query34	15.44	4.03	3.36
query35	3.39	3.34	3.35
query36	0.56	0.48	0.43
query37	0.09	0.07	0.06
query38	0.04	0.04	0.03
query39	0.03	0.03	0.03
query40	0.17	0.14	0.15
query41	0.08	0.03	0.02
query42	0.04	0.03	0.03
query43	0.04	0.03	0.03
Total cold run time: 95.3 s
Total hot run time: 24.49 s

[morningman]

run buildall

[hello-stephen]

TPC-H: Total hot run time: 29171 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 430a60634f41d0c45efa0a9e018c1c09de8f4407, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17574	5718	5658	5658
q2	2086	313	254	254
q3	10260	1464	846	846
q4	4687	484	346	346
q5	7432	827	527	527
q6	207	190	155	155
q7	810	835	618	618
q8	9250	1526	1221	1221
q9	5860	4611	4596	4596
q10	6864	1276	1000	1000
q11	453	253	236	236
q12	625	423	293	293
q13	18067	3205	2388	2388
q14	273	261	242	242
q15	q16	744	721	674	674
q17	1333	757	596	596
q18	7125	6246	6145	6145
q19	1112	1105	621	621
q20	389	354	220	220
q21	5173	2272	2436	2272
q22	371	314	263	263
Total cold run time: 100695 ms
Total hot run time: 29171 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	6362	6320	6305	6305
q2	708	551	543	543
q3	5226	5409	4907	4907
q4	2092	2227	1550	1550
q5	5263	5640	5682	5640
q6	291	210	170	170
q7	2307	2004	1806	1806
q8	3223	2833	2880	2833
q9	8211	8171	8423	8171
q10	3875	3729	3364	3364
q11	691	443	407	407
q12	742	771	583	583
q13	3014	3335	2581	2581
q14	313	326	291	291
q15	q16	727	715	659	659
q17	8578	7609	7435	7435
q18	13084	12220	13079	12220
q19	1032	847	834	834
q20	2198	2183	1921	1921
q21	6019	4672	5098	4672
q22	495	449	411	411
Total cold run time: 74451 ms
Total hot run time: 67303 ms

[hello-stephen]

TPC-DS: Total hot run time: 152300 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 430a60634f41d0c45efa0a9e018c1c09de8f4407, data reload: false

query5	4329	586	484	484
query6	450	207	195	195
query7	4815	440	224	224
query8	323	192	169	169
query9	8725	4015	4002	4002
query10	487	326	273	273
query11	5997	3471	3185	3185
query12	152	98	95	95
query13	1250	447	330	330
query14	6558	4797	4500	4500
query14_1	4371	4249	4266	4249
query15	208	209	203	203
query16	950	438	436	436
query17	873	678	554	554
query18	2422	427	313	313
query19	194	170	144	144
query20	81	80	78	78
query21	229	134	119	119
query22	12983	12915	12861	12861
query23	13216	12314	12098	12098
query23_1	12218	12075	12237	12075
query24	7105	822	466	466
query24_1	483	474	479	474
query25	528	419	367	367
query26	1267	250	148	148
query27	2765	435	274	274
query28	4611	1930	1913	1913
query29	1552	597	455	455
query30	299	220	181	181
query31	887	775	650	650
query32	152	90	93	90
query33	513	310	283	283
query34	888	836	519	519
query35	755	756	639	639
query36	804	823	735	735
query37	137	111	91	91
query38	1792	1773	1678	1678
query39	717	697	686	686
query39_1	699	678	672	672
query40	228	117	98	98
query41	71	68	69	68
query42	89	95	84	84
query43	369	380	329	329
query44	1328	690	696	690
query45	181	179	174	174
query46	888	993	562	562
query47	2956	2954	2953	2953
query48	318	318	263	263
query49	562	405	304	304
query50	669	271	202	202
query51	10169	10220	10439	10220
query52	78	80	70	70
query53	188	211	152	152
query54	262	192	173	173
query55	73	72	66	66
query56	229	207	227	207
query57	1619	1617	1597	1597
query58	284	250	247	247
query59	2352	2379	2128	2128
query60	283	256	219	219
query61	150	149	152	149
query62	388	346	295	295
query63	200	166	162	162
query64	2754	995	798	798
query65	3423	3382	3382	3382
query66	1802	420	304	304
query67	20114	20458	20288	20288
query68	3024	978	614	614
query69	388	294	251	251
query70	924	835	846	835
query71	288	229	207	207
query72	2739	2503	2254	2254
query73	517	526	301	301
query74	4538	4432	4278	4278
query75	2329	2287	1952	1952
query76	2335	1019	619	619
query77	357	407	297	297
query78	9035	8999	8527	8527
query79	960	901	510	510
query80	535	423	338	338
query81	535	316	279	279
query82	578	133	114	114
query83	304	198	186	186
query84	306	123	102	102
query85	819	432	366	366
query86	313	238	230	230
query87	1995	1974	1816	1816
query88	3649	2701	2658	2658
query89	332	296	257	257
query90	1922	179	179	179
query91	163	150	123	123
query92	94	88	90	88
query93	985	974	550	550
query94	444	304	287	287
query95	598	339	395	339
query96	660	516	230	230
query97	2427	2425	2298	2298
query98	154	149	150	149
query99	770	763	642	642
Total cold run time: 233628 ms
Total hot run time: 152300 ms

[hello-stephen]

ClickBench: Total hot run time: 24.32 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 430a60634f41d0c45efa0a9e018c1c09de8f4407, data reload: false

query1	0.01	0.00	0.01
query2	0.10	0.05	0.04
query3	0.27	0.10	0.12
query4	1.62	0.12	0.13
query5	0.25	0.22	0.24
query6	1.16	0.69	0.67
query7	0.04	0.01	0.00
query8	0.06	0.04	0.04
query9	0.42	0.36	0.35
query10	0.62	0.57	0.59
query11	0.20	0.15	0.13
query12	0.19	0.15	0.15
query13	0.48	0.50	0.49
query14	0.98	0.95	0.96
query15	0.65	0.60	0.60
query16	0.36	0.37	0.32
query17	1.14	1.13	1.15
query18	0.23	0.20	0.21
query19	2.09	1.99	1.98
query20	0.02	0.02	0.01
query21	15.49	0.28	0.15
query22	4.56	0.05	0.06
query23	16.27	0.32	0.12
query24	3.20	0.64	0.41
query25	0.13	0.05	0.06
query26	1.68	0.24	0.16
query27	0.04	0.03	0.04
query28	2.13	0.63	0.42
query29	12.54	4.14	3.30
query30	0.29	0.14	0.12
query31	2.77	0.59	0.33
query32	3.22	0.61	0.50
query33	3.32	3.14	3.22
query34	15.54	3.94	3.39
query35	3.34	3.34	3.31
query36	0.55	0.44	0.43
query37	0.09	0.06	0.07
query38	0.06	0.03	0.03
query39	0.03	0.04	0.03
query40	0.17	0.15	0.14
query41	0.08	0.03	0.03
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 96.46 s
Total hot run time: 24.32 s

[hello-stephen]

FE Regression Coverage Report

Increment line coverage 78.18% (129/165) 🎉
Increment coverage report
Complete coverage report

[morningman]

run buildall

[hello-stephen]

TPC-H: Total hot run time: 28934 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bdedd760765a9b65caa2a2f08a7f0b879f08b166, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17585	5697	5547	5547
q2	2117	321	257	257
q3	10230	1381	844	844
q4	4680	474	351	351
q5	7438	803	552	552
q6	195	192	153	153
q7	760	826	615	615
q8	9250	1362	1149	1149
q9	5690	4473	4456	4456
q10	6814	1292	1026	1026
q11	442	255	234	234
q12	632	415	294	294
q13	18040	3064	2381	2381
q14	283	270	245	245
q15	q16	749	730	672	672
q17	1191	733	567	567
q18	7149	6265	6215	6215
q19	1107	971	598	598
q20	397	332	234	234
q21	5492	2281	2388	2281
q22	364	310	263	263
Total cold run time: 100605 ms
Total hot run time: 28934 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	6128	6109	6145	6109
q2	667	591	570	570
q3	5313	5415	4982	4982
q4	2021	2172	1545	1545
q5	5260	5366	5992	5366
q6	265	210	158	158
q7	2358	2014	1945	1945
q8	3150	2794	2713	2713
q9	8372	8184	8459	8184
q10	3786	3753	3432	3432
q11	635	499	407	407
q12	736	735	541	541
q13	2857	3183	2496	2496
q14	307	334	312	312
q15	q16	705	747	657	657
q17	8423	7500	7470	7470
q18	13158	12281	13032	12281
q19	942	794	817	794
q20	2199	2195	1929	1929
q21	6046	4818	4941	4818
q22	510	490	418	418
Total cold run time: 73838 ms
Total hot run time: 67127 ms

[hello-stephen]

TPC-DS: Total hot run time: 151851 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bdedd760765a9b65caa2a2f08a7f0b879f08b166, data reload: false

query5	4319	619	460	460
query6	441	216	205	205
query7	4809	440	230	230
query8	324	187	172	172
query9	8776	3961	3923	3923
query10	488	305	296	296
query11	5966	3402	3192	3192
query12	150	88	86	86
query13	1251	439	326	326
query14	6572	4863	4509	4509
query14_1	4271	4284	4219	4219
query15	211	205	192	192
query16	974	493	453	453
query17	867	665	561	561
query18	2408	434	319	319
query19	207	173	141	141
query20	82	80	78	78
query21	207	135	117	117
query22	12959	12968	12821	12821
query23	13277	12449	12207	12207
query23_1	12345	12167	12344	12167
query24	7110	835	467	467
query24_1	474	466	467	466
query25	536	420	370	370
query26	1295	257	141	141
query27	2779	438	275	275
query28	4579	1928	1905	1905
query29	1591	605	463	463
query30	300	230	181	181
query31	875	785	656	656
query32	141	95	90	90
query33	526	316	253	253
query34	901	853	508	508
query35	767	774	644	644
query36	861	804	738	738
query37	133	102	93	93
query38	1806	1778	1681	1681
query39	728	703	697	697
query39_1	664	651	671	651
query40	212	116	100	100
query41	67	61	60	60
query42	84	82	85	82
query43	362	375	324	324
query44	1309	690	686	686
query45	180	177	169	169
query46	846	960	563	563
query47	2943	3024	2784	2784
query48	297	310	213	213
query49	576	393	292	292
query50	690	285	205	205
query51	10379	10480	10448	10448
query52	79	79	76	76
query53	185	223	160	160
query54	227	201	179	179
query55	74	69	63	63
query56	241	206	198	198
query57	1628	1597	1556	1556
query58	285	261	249	249
query59	2361	2406	2151	2151
query60	275	243	220	220
query61	145	149	142	142
query62	395	350	284	284
query63	192	161	160	160
query64	2740	927	775	775
query65	3444	3367	3374	3367
query66	1764	409	312	312
query67	20001	20082	19880	19880
query68	3010	991	602	602
query69	382	284	239	239
query70	888	828	839	828
query71	309	228	207	207
query72	2662	2537	2277	2277
query73	528	527	294	294
query74	4627	4446	4288	4288
query75	2337	2305	1917	1917
query76	2311	1025	596	596
query77	348	396	299	299
query78	9108	9035	8510	8510
query79	999	833	517	517
query80	1216	432	347	347
query81	574	319	278	278
query82	599	135	110	110
query83	286	191	175	175
query84	304	124	99	99
query85	838	421	364	364
query86	395	256	224	224
query87	2043	2003	1829	1829
query88	3601	2697	2660	2660
query89	345	294	263	263
query90	1917	178	179	178
query91	158	157	128	128
query92	100	87	91	87
query93	986	985	558	558
query94	650	301	273	273
query95	596	354	396	354
query96	662	517	232	232
query97	2446	2429	2331	2331
query98	156	146	140	140
query99	772	764	641	641
Total cold run time: 234760 ms
Total hot run time: 151851 ms

[hello-stephen]

ClickBench: Total hot run time: 24.46 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit bdedd760765a9b65caa2a2f08a7f0b879f08b166, data reload: false

query1	0.01	0.01	0.01
query2	0.11	0.05	0.06
query3	0.26	0.12	0.11
query4	1.61	0.13	0.13
query5	0.28	0.25	0.25
query6	1.18	0.68	0.69
query7	0.04	0.01	0.00
query8	0.05	0.03	0.04
query9	0.41	0.36	0.36
query10	0.61	0.58	0.57
query11	0.22	0.14	0.14
query12	0.18	0.14	0.15
query13	0.51	0.50	0.51
query14	0.96	0.97	0.95
query15	0.65	0.62	0.61
query16	0.31	0.32	0.31
query17	1.15	1.11	1.17
query18	0.23	0.21	0.21
query19	2.09	2.00	2.01
query20	0.02	0.02	0.01
query21	15.51	0.28	0.14
query22	4.45	0.06	0.05
query23	16.26	0.33	0.12
query24	3.27	0.62	0.40
query25	0.11	0.08	0.05
query26	1.28	0.26	0.14
query27	0.05	0.04	0.02
query28	2.05	0.59	0.40
query29	12.52	4.14	3.25
query30	0.28	0.13	0.13
query31	2.77	0.57	0.34
query32	3.22	0.62	0.52
query33	3.14	3.20	3.21
query34	15.49	4.07	3.39
query35	3.36	3.34	3.34
query36	0.54	0.44	0.45
query37	0.09	0.06	0.06
query38	0.04	0.04	0.04
query39	0.04	0.03	0.04
query40	0.17	0.15	0.14
query41	0.08	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.03	0.03
Total cold run time: 95.68 s
Total hot run time: 24.46 s

[hello-stephen]

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	65.10% (30803/47313)
Line Coverage	49.85% (323312/648553)
Region Coverage	45.34% (261370/576471)
Branch Coverage	46.99% (122541/260778)

[hello-stephen]

FE UT Coverage Report

Increment line coverage 90.96% (161/177) 🎉
Increment coverage report
Complete coverage report

[hello-stephen]

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	76.71% (35122/45788)
Line Coverage	61.90% (396893/641142)
Region Coverage	58.40% (335273/574128)
Branch Coverage	59.24% (154007/259979)

[hello-stephen]

FE Regression Coverage Report

Increment line coverage 79.10% (140/177) 🎉
Increment coverage report
Complete coverage report