Skip to content

SOLR-18312: introduce dedicated thread pool executor for httpClientBu… - #4655

Open
renatoh wants to merge 5 commits into
apache:mainfrom
renatoh:SOLR-18312-fix-Thread-Pool-Starvation-in-HttpJdkSolrClient
Open

SOLR-18312: introduce dedicated thread pool executor for httpClientBu…#4655
renatoh wants to merge 5 commits into
apache:mainfrom
renatoh:SOLR-18312-fix-Thread-Pool-Starvation-in-HttpJdkSolrClient

Conversation

@renatoh

@renatoh renatoh commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR (SOLR-18312) updates HttpJdkSolrClient to use a dedicated executor for the JDK HttpClient, aiming to prevent deadlocks when request bodies are streamed (producer) while the JDK client consumes them (consumer), particularly under HTTP/1.1 with concurrent requests.

Changes:

  • Introduce a separate httpClientExecutor that is always owned and shut down by HttpJdkSolrClient, instead of reusing the request-body streaming executor.
  • Add a concurrency regression test that issues multiple concurrent JsonQueryRequest calls with large streamed bodies while forcing HTTP/1.1.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File Description
solr/solrj/src/java/org/apache/solr/client/solrj/impl/HttpJdkSolrClient.java Adds a dedicated executor for the underlying JDK HttpClient and ensures it is shut down on close.
solr/solrj/src/test/org/apache/solr/client/solrj/impl/HttpJdkSolrClientTest.java Adds a concurrent, streamed-body test intended to catch deadlocks under HTTP/1.1.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +655 to +660
() -> {
JsonQueryRequest q = buildLargeBodyQuery();
try {
q.process(client);
} catch (Exception ignored) {
}
Comment on lines +664 to +665
CompletableFuture.allOf(futures.toArray(new CompletableFuture<?>[0]))
.get(45, TimeUnit.SECONDS);
Comment on lines +129 to +137
this.httpClientExecutor =
new ExecutorUtil.MDCAwareThreadPoolExecutor(
0,
Integer.MAX_VALUE,
60,
TimeUnit.SECONDS,
new SynchronousQueue<>(),
new SolrNamedThreadFactory(this.getClass().getSimpleName() + "-http"));
httpClientBuilder.executor(this.httpClientExecutor);

@dsmiley dsmiley left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it'd be more natural for the builder/configured executor to be for the httpClient usage itself, just as the jetty one.

@dsmiley

dsmiley commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

I pushed an improvement to defer the use of the executor for body thread pushing to only start when it's requested (by the supplier). Maybe in practice won't be notice but I like the code for it much better now. CC @serhiy-bzhezytskyy

@serhiy-bzhezytskyy

Copy link
Copy Markdown
Contributor

@dsmiley the deferral does more than tidy the code — it removes the leak by construction for one whole class of failure. I measured it with a counting executor on a refused connection: before e534b380e one body task is submitted and then has to be released; after it, zero (expected:<0> but was:<1> against the previous commit). So the original SOLR-17707 guard now only covers "body requested, then the connection dropped", which is the narrower case it should have been.

Two things I checked while I was in here, both fine:

  • the mechanism sentence still holds. I mutated the fix on main to keep only cancel(true) and the SOLR-17707 test fails with content-writing thread leaked, still blocked after failure — so close(sink) is still load-bearing, and moving the comment along with the code was right.
  • contentWritingSink is assigned before contentWritingFuture, so a releaseContentWriting() arriving between them finds a null future and an already-closed sink. Over 200 runs that is 200 writers exiting on IOException with the pool fully drained — log noise, not a leak. Not worth a change.

The one thing missing is a test for the new property. I have the one above if you want it, against this branch or as a follow-up.

@dsmiley

dsmiley commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Thanks for the review Serhiy. testStuckContentWritingThreadIsReleasedOnFailure that you added previously is good enough IMO. I just pushed a change to switch from volatile to synchronized, which avoids the sequence issue you pointed out -- thanks for that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants