Overhaul map JMH benchmarks: remove thread contention and split by use case by dougqh · Pull Request #11679 · DataDog/dd-trace-java

dougqh · 2026-06-18T17:40:09Z

What This Does

Overhauls the internal-api map microbenchmarks so each one isolates the dimension it actually claims to measure, and splits the general map benchmark by use case.

TagMapAccessBenchmark — removes cross-thread contention: all shared state is immutable, while every bit of mutable state (the pre-populated read map, lookup index, reader flyweight) lives in a @State(Scope.Thread) holder, so threads never contend on a shared map. Now runs at @Fork(2) to match the rest.
Splits UnsynchronizedMapBenchmark into two classes. The correct JMH @State scope differs by use case and can't vary by @Param, so one class can't host both threading models:
- ImmutableMapBenchmark — precomputed / read-mostly maps shared across threads (@State(Scope.Benchmark)); sharing is realistic and contention-free because nothing mutates after construction. get / iterate across HashMap, LinkedHashMap, TreeMap, TagMap.
- SingleThreadedMapBenchmark — per-thread mutable lifecycle (@State(Scope.Thread)): create / clone + reads. Adds a Collections.synchronizedMap case to measure the uncontended synchronization tax — each thread owns its synchronized map, so the monitor is only ever locked by one thread (bias never revoked). The unsynchronized HashMap get/iterate are the in-harness baseline; the tax is the delta.
Fixes a latent bug: the old iterate_linkedHashMap iterated TREE_MAP.

Motivation

Before making further changes to TagMap, I want robust benchmarks in place. The previous shared mutable state (a cross-thread counter/index, shared maps) turned several "iteration / lookup" benchmarks into contention measurements rather than measurements of the map itself. Isolating state per thread — and separating the read-mostly-shared use case from the single-threaded-mutable one — fixes that.

Notes

Run at default JVM flags (what customers run). The synchronized-map result is meant to be read across JVM versions: Java ≤ 11 has biased locking on by default (uncontended same-thread locking ≈ free), Java 15+ has it disabled (JEP 374), so a Java 11 → 17 comparison shows the biased-locking cliff directly.
Result blocks are intentionally empty pending a fresh multi-JVM run.
The -Pjmh.includes / -PtestJvm gradle wiring was moved to its dedicated PR (Wire -Pjmh.includes and -PtestJvm into internal-api JMH config #11703); this PR no longer touches build.gradle.kts.
Independent of the LegacyTagMap removal (Remove LegacyTagMap; OptimizedTagMap is the sole TagMap implementation #11678) — uses only the public TagMap API, so it builds/runs on master and on that branch alike.
First of a planned per-span-type benchmark suite (web/db/queue/…), giving instrumentation authors relatable numbers and the natural vehicle for per-type optimizations later.

🤖 Generated with Claude Code

@threads

Throughput microbenchmark for TagMap insert/getObject/getEntry over a representative HTTP-server tag set. All mutable state (the read map) lives in @State(Scope.Thread) so @threads(8) runs measure TagMap rather than cross-thread contention on a shared map/counter/flyweight — the flaw that made earlier TagMap benchmarks misleading. Run config is baked into annotations (the me.champeau.jmh plugin ignores -Pjmh.* flags). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

dd-octo-sts · 2026-06-18T18:02:18Z

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	13.97 s	13.91 s	[-0.6%; +1.5%] (no difference)
startup:insecure-bank:tracing:Agent	12.88 s	12.90 s	[-0.8%; +0.5%] (no difference)
startup:petclinic:appsec:Agent	16.29 s	16.71 s	[-6.9%; +1.9%] (no difference)
startup:petclinic:iast:Agent	16.76 s	15.94 s	[-0.9%; +11.2%] (unstable)
startup:petclinic:profiling:Agent	16.72 s	16.77 s	[-1.1%; +0.5%] (no difference)
startup:petclinic:sca:Agent	16.39 s	16.76 s	[-6.3%; +1.9%] (no difference)
startup:petclinic:tracing:Agent	15.97 s	16.13 s	[-2.1%; +0.1%] (no difference)

Commit: a9329e0b · CI Pipeline · Benchmarking Platform UI

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

dougqh · 2026-06-22T19:28:27Z

+public class TagMapAccessBenchmark {
+  // a representative HTTP-server-ish tag set (immutable -> safe to share across threads)
+  static final String[] NAMES = {
+    "http.request.method",


Future intended changes will care about the specifics of the tags, so using real tags is preferable for future-proofing

sharedLookupIndex was a plain static int incremented by all 8 JMH threads without synchronization — a data race that turned the get benchmarks into a contention measurement rather than a map measurement. Move the index to @State(Scope.Thread) so each thread has its own cursor, matching the approach used in TagMapAccessBenchmark. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Without this, -Pjmh.includes is silently ignored by the me.champeau.jmh plugin, requiring a full fat-jar build to run a single benchmark. -PtestJvm was also ignored for JMH execution, defaulting to the Gradle daemon JVM regardless of the requested version. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Re-run after fixing the shared-index data race, on Java 17 with correct per-thread scaffolding state. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

The '2x faster construction' claim was stale — Java 17 numbers show ~40%. Also clarifies that LinkedHashMap's cost is purely at construction; gets and iteration are equivalent to HashMap. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…sBenchmark Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Align TagMapAccessBenchmark with UnsynchronizedMapBenchmark at @fork(2) for steadier numbers (results to be refreshed on the next run). Also revert the internal-api/build.gradle.kts -Pjmh.includes / -PtestJvm wiring, which belongs in its dedicated PR (#11703), not here. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…Benchmark Replace UnsynchronizedMapBenchmark with two classes that each pick the correct threading model for their use case (the @State scope can't vary by @Param, so one class can't host both): - ImmutableMapBenchmark: precomputed/read-mostly maps shared across threads (@State(Scope.Benchmark)) -- sharing is correct since read-only. get/iterate across HashMap, LinkedHashMap, TreeMap, TagMap. - SingleThreadedMapBenchmark: per-thread mutable lifecycle (@State(Scope.Thread)). create/clone + reads. Adds a Collections.synchronizedMap case to measure the uncontended synchronization tax (per-thread => bias never revoked); the unsynchronized HashMap get/iterate are the in-harness baseline. The biased- locking effect shows when comparing across JVM versions at stock flags. Also fixes a latent bug in the old iterate_linkedHashMap, which iterated TREE_MAP. Stale result blocks dropped; numbers pending a fresh multi-JVM run. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…edMapBenchmark StringIndex's benchmark integration is moving to the dedicated benchmark PRs (set overhaul #11721, map overhaul #11679) and will be folded in there later. Revert both benchmark files to master so this PR is purely the StringIndex data structure + tests. Avoids the #11679/#11721 deletions-vs-edits conflicts too. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

- Add a Map.copyOf case (via CollectionUtils.tryMakeImmutableMap -> JDK MapN) to ImmutableMapBenchmark: get / get_sameKey / iterate. MapN is the agent's actual fixed-config-map representation and the honest immutable-map baseline. - Fix TagMapAccessBenchmark's @link to the deleted UnsynchronizedMapBenchmark -> SingleThreadedMapBenchmark (which now holds the clone cases). - Note that interned (_sameKey) lookups are the common tracer case (keys are typically interned tag-name constants). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…s-benchmark

bric3 · 2026-06-24T10:02:38Z

      tagMap.set(INSERTION_KEYS[i], i); // primitive support
    }
+    // JDK compact immutable map (MapN on Java 10+); the agent's actual fixed-map representation.
+    copyOfMap = CollectionUtils.tryMakeImmutableMap(hashMap);


quibble: Maybe there's a better name than copyOf, maybe tracerImmautableMap

dougqh added type: enhancement Enhancements and improvements comp: core Tracer core tag: no release notes Changes to exclude from release notes tag: ai generated Largely based on code generated by an AI or LLM labels Jun 18, 2026

This comment has been minimized.

Sign in to view

dougqh marked this pull request as ready for review June 22, 2026 19:27

dougqh requested a review from a team as a code owner June 22, 2026 19:27

dougqh requested a review from ygree June 22, 2026 19:27

dougqh commented Jun 22, 2026

View reviewed changes

dougqh and others added 6 commits June 22, 2026 16:58

Update UnsynchronizedMapBenchmark results with Java 17 numbers

9dc9122

Re-run after fixing the shared-index data race, on Java 17 with correct per-thread scaffolding state. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Add builder-style insert benchmarks and update results in TagMapAcces…

cd883e1

…sBenchmark Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Merge branch 'master' into dougqh/tagmap-access-benchmark

02d08f0

dougqh changed the title ~~Add a threading-correct TagMap access microbenchmark~~ Removing thread contention from Unsynchronized Map Benchmarks Jun 23, 2026

dougqh and others added 2 commits June 23, 2026 14:27

dougqh changed the title ~~Removing thread contention from Unsynchronized Map Benchmarks~~ Overhaul map JMH benchmarks: remove thread contention and split by use case Jun 23, 2026

dougqh mentioned this pull request Jun 23, 2026

Overhaul set benchmarks: split Immutable / SingleThreaded, add Set.copyOf #11721

Draft

dougqh and others added 2 commits June 23, 2026 18:03

Merge remote-tracking branch 'origin/master' into dougqh/tagmap-acces…

a9329e0

…s-benchmark

bric3 approved these changes Jun 24, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Overhaul map JMH benchmarks: remove thread contention and split by use case#11679

Overhaul map JMH benchmarks: remove thread contention and split by use case#11679
dougqh wants to merge 11 commits into
masterfrom
dougqh/tagmap-access-benchmark

dougqh commented Jun 18, 2026 •

edited

Loading

Uh oh!

This comment has been minimized.

dd-octo-sts Bot commented Jun 18, 2026 •

edited

Loading

Uh oh!

dougqh Jun 22, 2026

Uh oh!

bric3 Jun 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

dougqh commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What This Does

Motivation

Notes

Uh oh!

This comment has been minimized.

dd-octo-sts Bot commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🟢 Java Benchmark SLOs — All performance SLOs passed

Uh oh!

dougqh Jun 22, 2026

Choose a reason for hiding this comment

Uh oh!

bric3 Jun 24, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

dougqh commented Jun 18, 2026 •

edited

Loading

dd-octo-sts Bot commented Jun 18, 2026 •

edited

Loading