SOLR-18312: introduce dedicated thread pool executor for httpClientBu… - #4655
SOLR-18312: introduce dedicated thread pool executor for httpClientBu…#4655renatoh wants to merge 5 commits into
Conversation
There was a problem hiding this comment.
Pull request overview
This PR (SOLR-18312) updates HttpJdkSolrClient to use a dedicated executor for the JDK HttpClient, aiming to prevent deadlocks when request bodies are streamed (producer) while the JDK client consumes them (consumer), particularly under HTTP/1.1 with concurrent requests.
Changes:
- Introduce a separate
httpClientExecutorthat is always owned and shut down byHttpJdkSolrClient, instead of reusing the request-body streaming executor. - Add a concurrency regression test that issues multiple concurrent
JsonQueryRequestcalls with large streamed bodies while forcing HTTP/1.1.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| solr/solrj/src/java/org/apache/solr/client/solrj/impl/HttpJdkSolrClient.java | Adds a dedicated executor for the underlying JDK HttpClient and ensures it is shut down on close. |
| solr/solrj/src/test/org/apache/solr/client/solrj/impl/HttpJdkSolrClientTest.java | Adds a concurrent, streamed-body test intended to catch deadlocks under HTTP/1.1. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| () -> { | ||
| JsonQueryRequest q = buildLargeBodyQuery(); | ||
| try { | ||
| q.process(client); | ||
| } catch (Exception ignored) { | ||
| } |
| CompletableFuture.allOf(futures.toArray(new CompletableFuture<?>[0])) | ||
| .get(45, TimeUnit.SECONDS); |
| this.httpClientExecutor = | ||
| new ExecutorUtil.MDCAwareThreadPoolExecutor( | ||
| 0, | ||
| Integer.MAX_VALUE, | ||
| 60, | ||
| TimeUnit.SECONDS, | ||
| new SynchronousQueue<>(), | ||
| new SolrNamedThreadFactory(this.getClass().getSimpleName() + "-http")); | ||
| httpClientBuilder.executor(this.httpClientExecutor); |
dsmiley
left a comment
There was a problem hiding this comment.
I think it'd be more natural for the builder/configured executor to be for the httpClient usage itself, just as the jetty one.
|
I pushed an improvement to defer the use of the executor for body thread pushing to only start when it's requested (by the supplier). Maybe in practice won't be notice but I like the code for it much better now. CC @serhiy-bzhezytskyy |
|
@dsmiley the deferral does more than tidy the code — it removes the leak by construction for one whole class of failure. I measured it with a counting executor on a refused connection: before Two things I checked while I was in here, both fine:
The one thing missing is a test for the new property. I have the one above if you want it, against this branch or as a follow-up. |
|
Thanks for the review Serhiy. |
see ticket https://issues.apache.org/jira/browse/SOLR-18312