[geom] Reduce thread false sharing and improve multithreaded TGeo navigation - #22955
Open
sawenzel wants to merge 2 commits into
Open
[geom] Reduce thread false sharing and improve multithreaded TGeo navigation#22955sawenzel wants to merge 2 commits into
sawenzel wants to merge 2 commits into
Conversation
added 2 commits
July 29, 2026 14:33
…igation
This commit improves multithreaded TGeo navigation. The changes come from profiling
a parallel geometry scan in ALICE: filling the material budget LUT on 28 cores now
scales from 12× to 23× speedup (139 s -> 72 s).
Two costs dominated. Every per-thread scratch lookup went through
TGeoManager::ThreadId(), a non-inlined cross-library __tls_get_addr call paid on
every boolean, section, and division query. In addition, the per-object
ThreadData_t blocks were allocated back-to-back, so slots belonging to different
threads often shared cache lines and invalidated each other on every write
(false sharing), especially across sockets.
Each object now gets a dense index, while the per-thread state lives in a single
thread_local vector indexed by it. GetThreadData() becomes a header-inlined TLS
read followed by an indexed load, and each thread owns its entire vector.
Main changes:
* TGeoBoolNode, TGeoVolumeAssembly, TGeoPatternFinder, TGeoPgon, TGeoXtru
now use indexed thread-local storage.
* TGeoPgon/TGeoXtru: add noexcept move constructors for ThreadData_t (which
owns heap buffers, and for TGeoXtru also a TGeoPolygon aliasing them), so
entries remain valid when the vector grows.
* TGeoPatternFinder: reuse the transformation matrix across generations. The
matrix is owned by the geometry manager and is never released, so creating
a new one would leak one matrix per (thread, finder).
Provisioning is no longer needed. ClearThreadData() now increments a generation
counter, and each thread lazily rebuilds its slot on first access. As a result,
CreateThreadData() and SetMaxThreads() are no longer required to support a given
thread count: any number of threads now works out of the box.
The generation counters are atomic because ClearThreadData() is const and may be
called concurrently.
Assisted-by: Claude Code (review, hardening and benchmarking)
Supervised-by: Sandro Wenzel <sandro.wenzel@cern.ch>
This commit provides a test exercising multithreaded TGeo navigation. Eight threads navigating the same geometry must give exactly the single-threaded answer. Covers TGeoXtru, TGeoPgon, TGeoVolumeAssembly, TGeoBoolNode (composite shape) and TGeoPatternFinder (divided volume). The threads book their navigators lazily, so it also exercises AddNavigator() against the navigator-map readers. Assisted-by: Claude Code (review, hardening and benchmarking) Supervised-by: Sandro Wenzel <sandro.wenzel@cern.ch>
Contributor
Author
This graph demonstrates the improved scaling. It was measured on an older version of ROOT but it translates to the master version. |
Test Results 23 files 23 suites 3d 18h 56m 28s ⏱️ Results for commit 9ebb0b7. |
silverweed
reviewed
Jul 30, 2026
| /// Each thread owns its whole vector, so no two threads ever write the same cache line. | ||
| ThreadData_t &GetThreadData() const | ||
| { | ||
| thread_local std::vector<ThreadData_t> tdata; |
Contributor
There was a problem hiding this comment.
Are all these vectors simply never freed? Probably not a problem for most threads, but is this ever called from the main thread? If so the memory may be hoarded indefinitely for the entire process duration, no?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR improves the multithreaded performance of TGeoNavigator by reducing the overhead of thread-local scratch data access and eliminating false sharing between threads.
The changes are the result of profile-guided optimization while profiling the parallel material budget scan in ALICE (see AliceO2Group/AliceO2#15641).
On a 28-core machine, the material budget LUT generation improves from a 12× to a 23× speedup (139 s → 72 s). Since the optimizations are in the geometry navigation infrastructure itself, they should benefit any application relying on multithreaded geometry algorithms.
This is the result of a summer internship of @trwenz on whose behalf the PR is opened.