Liveness epoch/urgency fixes, deferred chain invalidation, and debug JNI test seams by jbachorik · Pull Request #798 · DataDog/java-profiler

jbachorik · 2026-09-17T17:26:00Z

What does this PR do?:

Fixes and debug tooling for the reference-chain engine and the liveness tracker. The lifecycle/JFR/vmEntry wiring this PR first described landed in main via #797 and is not in this diff.

Debug JNI test seams. javaApi.cpp adds the ...ForTest natives declared in JavaProfiler.java: pending-expand inspection, pass/gate predicates, tracker internals. Debug builds only. Release builds export no-op stubs, so the calls do not throw UnsatisfiedLinkError.
Deferred resolved-chain invalidation. Heap callbacks queue chain evictions into a preallocated buffer; the BFS thread applies them in one batch outside the walk. No _resolved_chains_lock inside the FollowReferences pause. All callback paths defer: walk admission and auto-mark. The pending queue and its overflow flag reset at every recording and search boundary.
Liveness epoch accounting. cleanup_table() takes an account_epoch flag. The overflow sweep in track() no longer claims the GC epoch the background sweep needs. Entries store their admission epoch and age only for GCs they survived.
Urgency-boost gating. ObjectSampler::start() finishes LivenessTracker::start() before it enables SAMPLED_OBJECT_ALLOC or publishes _active. New counters LIVENESS_URGENT_BOOST_ADMITS / LIVENESS_URGENT_BOOST_BACKED_OFF. secondsToOOM() no longer divides by zero.

Motivation:

Review feedback on the reference-chain work. An overflow sweep could empty the chain cache before a dump. Urgency admissions raced table cleanup. Epoch aging undercounted entries. The JNI seams test engine state deterministically.

Additional Notes:

PriorityExpandSet::insert() returns false on a full table instead of probing forever. contains() keeps its unbounded probe; the set stays under half capacity. See the inline thread on referenceChains.h.
The test seams prime ProfiledThread TLS on entry, like the other JNI entry points.

How to test the change?:

:ddprof-lib:gtestDebug / :ddprof-lib:gtestRelease — referenceChains and liveness suites, incl. new tests for defer/drain, the PriorityExpandSet full-table case, and account_epoch aging.
Java integration tests via testDebug / testRelease.

For Datadog employees:

If this PR touches code that signs or publishes builds or packages, or handles
credentials of any kind, I've requested a security review (run the dd:platform-security-review
skill, or file a request via the PSEC review form).
bewaire also runs automatically on every PR.
This PR doesn't touch any of that.
JIRA: PROF-15341

dd-octo-sts · 2026-09-17T17:29:19Z

Scan-Build Report

User:	runner@runnervmlun5p
Working Directory:	/home/runner/work/java-profiler/java-profiler/ddprof-lib/src/test/make
Command Line:	make -j4 all
Clang Version:	Ubuntu clang version 18.1.3 (1ubuntu1)
Date:	Thu Sep 17 20:07:57 2026

Bug Summary

C++ move semantics
Bug Type	Quantity	Display?
All Bugs	4
Use-after-move	1
Logic error
Dereference of null pointer	1
Result of operation is garbage or undefined	1
Unused code
Dead increment	1

Reports

Bug Group	Bug Type ▾	File	Function/Method	Line	Path Length
Unused code	Dead increment	referenceChains.cpp	collectStaticFieldAnchorsForRotation	3677	1
Logic error	Dereference of null pointer	faultInjection.cpp	crashNow	24	2
Logic error	Result of operation is garbage or undefined	livenessTracker.cpp	secondsToOOM	1383	24
C++ move semantics	Use-after-move	referenceChains.cpp	buildCanaryChainEvent	6083	68

dd-octo-sts · 2026-09-17T17:44:56Z

CI Test Results

Run: #37957706969 | Commit: 32b7dbf | Duration: 23m 44s (longest job)

✅ All 32 test jobs passed

Status Overview

JDK	glibc-aarch64/debug	glibc-amd64/debug	musl-aarch64/debug	musl-amd64/debug
8	-	✅	-	-
8-ibm	-	✅	-	-
8-j9	✅	✅	-	-
8-librca	-	-	✅	✅
8-orcl	-	✅	-	-
11	-	✅	-	-
11-j9	✅	✅	-	-
11-librca	-	-	✅	✅
17	✅	✅	-	-
17-graal	✅	✅	-	-
17-j9	✅	✅	-	-
17-librca	-	-	✅	✅
21	✅	✅	-	-
21-graal	✅	✅	-	-
21-librca	-	-	✅	✅
25	✅	✅	-	-
25-graal	✅	✅	-	-
25-librca	-	-	✅	✅

Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled

Summary: Total: 32 | Passed: 32 | Failed: 0

Updated: 2026-10-09 16:40:54 UTC

dd-octo-sts · 2026-09-17T17:55:20Z

✅ All 40 integration tests passed

📊 Dashboard · 👷 Pipeline · 📦 5a1f7038

chatgpt-codex-connector · 2026-09-17T20:03:56Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-06T14:34:28.345419Z	`d21eb08`	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 93f2dbf486

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

datadog-datadog-prod-us1-2

Stopping a recording can omit all reference-chain events since the last dump. The lifecycle also keeps strong thread references and stale chain data across recording sessions.

_{Open Bits AI session}

_{🤖 Datadog Autotest · Commit 93f2dbf · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}

rkennke

Thanks, the fixes in 33686e9 look good. The deferral now covers the callback eviction path, the drain hands off the queue with an O(1) swap, the pending buffers are reset at every boundary, resetSession() closes the stale-flags gap, and the new tests cover what I asked for. (And you're right about _table_size, it's int; my nit was wrong.)

A few things still need sorting before this can merge:

Review tool output got committed. address-responses.json, preflight-evaluations.json, preflight-feedback.json, reviews.json and threads.json were added at the repo root. Please drop them from the branch.
Spec docs. The commit also adds docs/sphinx/specs/2026-10-09-deferred-chain-invalidation-correctness.md. I'm not sure we want spec docs in the repo at all: they describe a point-in-time plan, go stale as soon as the code moves on, and the reasoning belongs in the PR description or in the code comments. I'd leave this one out of the PR. There are already two in main (docs/sphinx/specs/2026-05-29-... and 2026-10-05-...); we should decide separately whether to keep those.
Merge conflict with main in ddprof-lib/src/main/cpp/profiler.h, so the PR can't be merged right now.
PR title/description still describe the lifecycle/JFR/vmEntry wiring that landed in #797 and the old ObjectSampler rollback. They should cover what this PR does now: the debug JNI seams, the liveness epoch/urgency/session fixes, and deferred chain invalidation.
CI on the new head was still running when I looked; I'll check again once it's done.

One small nit inline.

Introduces the ReferenceChainEvent/ReferenceChainAbandonedEvent payloads (event.h), their JFR metadata (jfrMetadata.*), and the FlightRecorder emission paths that serialize chain events into JFR recording buffers, including the constant-pool handling for per-hop edge labels. Emission is pull-style: profiler.cpp snapshots events and hands them to FlightRecorder; this layer does not depend on the tracker itself.

- ReferenceChainEvent carries one vector of ReferenceChainHop (klass id + retention-edge label) instead of two parallel vectors - Compress the sub-option floor/ceiling rationale and the provisional default constant comments to one concise statement each - Drop design-doc and Jira references from code comments; revert the unrelated LineNumberTable comment rewrite

- cleanup_table: account_epoch flag; track() overflow sweep skips epoch claim - admitForTracking: LIVENESS_URGENT_BOOST_ADMITS/BACKED_OFF counters - secondsToOOM: zero-denominator guard; ringThirdsStats -> ringWindowStats - JNI poll seam calls finishLoopIterationSerialized - gtests: DipThenRecover, OddLengthRisingWindow

…ckage-private

…rences pause - heapReferenceCallback ran invalidateResolvedChain() inside the FollowReferences STW pause, taking _resolved_chains_lock, which drainPendingChainEvents() holds across its cache copy on the JFR dump thread. Evictions are now queued and applied on the BFS thread outside any walk. - PriorityExpandSet::insert() had no occupancy bound and could spin forever on a full table; it now returns false.

… seam - referenceChainsPesTests.inc: PriorityExpandSet round-trip, rebuildFrom, full-table self-termination; ClassTagAllocator negative-tag minting - referenceChainsTestAccessors.h: cacheChain() passes source_tag and source_tag_val through unchanged - referenceChainsOomTests.inc: setCanaryBackoffForTest() takes ns

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

jbachorik · 2026-10-11T14:10:22Z

@rkennke I think I addressed all. I am planning to merge shortly if I don't hear any objections

kaahos

this looks good to me; thanks for the changes!

jbachorik added AI AI: Author Signed-off labels Sep 17, 2026

jbachorik mentioned this pull request Sep 17, 2026

C++ unit tests for reference-chain tracking #799

Closed

2 tasks done

jbachorik force-pushed the jb/rc-4-profiler-wiring branch from e8f5ee5 to 93f2dbf Compare September 17, 2026 19:08

jbachorik added this pull request to stack #803 September 17, 2026 20:02

jbachorik marked this pull request as ready for review September 17, 2026 20:03

jbachorik requested a review from a team as a code owner September 17, 2026 20:03

jbachorik marked this pull request as draft September 17, 2026 20:04

jbachorik force-pushed the jb/rc-4-profiler-wiring branch from 93f2dbf to 217a7ab Compare September 17, 2026 20:05

chatgpt-codex-connector Bot reviewed Sep 17, 2026

View reviewed changes

datadog-datadog-prod-us1-2 Bot reviewed Sep 17, 2026

View reviewed changes

Comment thread ddprof-lib/src/main/cpp/profiler.cpp

Comment thread ddprof-lib/src/main/cpp/profiler.cpp Outdated

Comment thread ddprof-lib/src/main/cpp/profiler.cpp Outdated

Comment thread ddprof-lib/src/main/cpp/profiler.cpp Outdated

Comment thread ddprof-lib/src/main/cpp/profiler.cpp Outdated

jbachorik force-pushed the jb/rc-4-profiler-wiring branch from 217a7ab to b5588b7 Compare September 17, 2026 20:19

jbachorik force-pushed the jb/rc-4-profiler-wiring branch from b5588b7 to b09e78d Compare September 17, 2026 20:20

jbachorik mentioned this pull request Sep 17, 2026

JFR event types and emission for reference chains #796

Merged

2 tasks done

jbachorik force-pushed the jb/rc-4-profiler-wiring branch 2 times, most recently from e99dbc2 to 61cfd3d Compare September 17, 2026 21:33

jbachorik force-pushed the jb/rc-4-profiler-wiring branch from 61cfd3d to ac29386 Compare September 17, 2026 22:15

This comment has been minimized.

Sign in to view

jbachorik force-pushed the jb/rc-4-profiler-wiring branch from 5f7ef97 to 2fbc79b Compare September 18, 2026 07:06

rkennke reviewed Oct 9, 2026

View reviewed changes

Comment thread ddprof-lib/src/main/cpp/referenceChainWalk.cpp Outdated

jbachorik and others added 20 commits October 9, 2026 16:44

Wire reference chains into the profiler lifecycle and JNI API

ba42eb3

Drop remaining Jira and cross-layer test references

c72dedd

misc: objectSampler start order, dump-failure test log, Java seams pa…

4f0ac15

…ckage-private

objectSampler: roll back on LivenessTracker start failure

39cd841

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

javaApi: release stubs for debug test seams, null-check jvmti

3854f88

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

referenceChains: batch deferred invalidations, cache probe start

c67a79b

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

tests: cover urgency backstop and account_epoch, tidy PES tests

57da0a2

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Trim comments and test-seam Javadocs

7ea2afa

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

sphinx: address review feedback on PR #798

96b2691

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

sphinx: address review feedback on PR #798

b1026ad

Trim review-fix comments

1ced450

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

sphinx: address review feedback on PR #798

aec7774

Fix urgency check comment

3f071a2

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Address deep review findings on reference chains

4ce119b

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

sphinx: address review feedback on PR #798

6c84483

jbachorik force-pushed the jb/rc-4-profiler-wiring branch from 33686e9 to 6c84483 Compare October 9, 2026 14:45

jbachorik changed the title ~~Wire reference chains into profiler lifecycle and JNI API~~ Liveness epoch/urgency fixes, deferred chain invalidation, and debug JNI test seams Oct 9, 2026

jbachorik added 2 commits October 9, 2026 17:12

Defer chain invalidation on the auto-mark path too

61177f0

Skip the terminal-outcome guard on TSan pass-budget exhaustion

5a1f703

kaahos approved these changes Oct 12, 2026

View reviewed changes

jbachorik merged commit 3dd5e40 into main Oct 12, 2026
163 checks passed

jbachorik deleted the jb/rc-4-profiler-wiring branch October 12, 2026 06:51

github-actions Bot added this to the 1.52.0 milestone Oct 12, 2026

Conversation

jbachorik commented Sep 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dd-octo-sts Bot commented Sep 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Scan-Build Report

Bug Summary

Reports

Uh oh!

dd-octo-sts Bot commented Sep 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CI Test Results

Status Overview

Uh oh!

dd-octo-sts Bot commented Sep 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chatgpt-codex-connector Bot commented Sep 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

datadog-datadog-prod-us1-2 Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment has been minimized.

rkennke left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jbachorik commented Oct 11, 2026

Uh oh!

kaahos left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jbachorik commented Sep 17, 2026 •

edited

Loading

dd-octo-sts Bot commented Sep 17, 2026 •

edited

Loading

dd-octo-sts Bot commented Sep 17, 2026 •

edited

Loading

dd-octo-sts Bot commented Sep 17, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Sep 17, 2026 •

edited

Loading