Repository navigation
[RUM-19398] Benchmarks: add cold start scenario - #1520
marco-saia-datadog wants to merge 3 commits into
Conversation
🎉 All green!🧪 All tests passed 🔗 Commit SHA: 1d0266b | Docs | View more details | Give us feedback! |
|
@codex review |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4fd5de2f45
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| setupFromTestConfig(DEFAULT_ENV_TEST_CONFIG); | ||
|
|
||
| consumeColdStartConfig().then(coldStartConfig => { | ||
| setupFromTestConfig(coldStartConfig); |
There was a problem hiding this comment.
When an armed coldStart is consumed, this routes it through setupFromTestConfig, which awaits startCollectingVitals before mounting ColdStartScenario. Both native implementations initialize the benchmark meter/exporter and observers during that call, so the process-start-to-layout TTFD includes benchmark instrumentation setup rather than only the application startup being compared. Skip or defer the regular vitals collector for this scenario so it cannot distort both baseline and instrumented results.
Useful? React with 👍 / 👎.
| clientToken: datadogConfig.clientToken, | ||
| applicationID: datadogConfig.applicationID, | ||
| apiKey: datadogConfig.apiKey, | ||
| env: datadogConfig.env, |
There was a problem hiding this comment.
When DD_SITE is anything other than US1, this native config drops datadogConfig.site, while both new native uploaders hardcode https://api.datadoghq.com. Consequently an EU, US3, US5, AP, or government-site API key is sent to the wrong intake and the cold-start metric is not recorded. Pass the configured site through and select the corresponding distribution intake.
Useful? React with 👍 / 👎.
| apiKey = config.getString("apiKey") ?: "", | ||
| run = config.getString("runType") ?: "", | ||
| scenario = config.getString("scenario") ?: "", | ||
| env = BuildConfig.BUILD_TYPE, |
There was a problem hiding this comment.
For Android benchmark builds, DD_ENV is passed into reportFullyDisplayed, but this code replaces it with BuildConfig.BUILD_TYPE. The CI configuration uses DD_ENV=dev while the uploaded application is typically a release build, so Android TTFD points are tagged env:release instead of env:dev and disappear from environment-scoped benchmark queries or dashboards. Use the env value from the native config, as the iOS implementation does.
Useful? React with 👍 / 👎.
| consumeColdStartConfig().then(coldStartConfig => { | ||
| setupFromTestConfig(coldStartConfig); |
There was a problem hiding this comment.
When BENCH_SCENARIO contains any valid scenario and a cold start is armed, the environment scenario is started immediately while this asynchronous consume starts the cold scenario independently. The first scenario can therefore mount and initialize the SDK before the cold scenario replaces it, and both calls also start separate native vitals meters; an instrumented cold-start sample then no longer measures a fresh SDK initialization. Consume the pending configuration first and choose it in preference to the environment default rather than starting both.
Useful? React with 👍 / 👎.
4fd5de2 to
1d0266b
Compare
What does this PR do?
Adds a
coldStartscenario to the benchmark app. It measures the time to fully displayed (TTFD) on cold start, withbaselineandinstrumentedrun types, on iOS and Android.baseline: aNavigationContainerwith one stack screen, showing static content without network calls, and no Datadog SDK.instrumented: the same app, wrapped inDatadogProvider(with the default verbosity, as in customer release builds) and with React Navigation view tracking. Once the first screen is laid out andonInitializationhas fired, the app callsDdRum.reportAppFullyDisplayed().One measurement for both run types. At the same point in the app, both run types call a new native method,
BenchmarkVitals.reportFullyDisplayed(). It measuresnow − process startwithout going through the SDK:sysctlkp_proc.p_starttimeActivePrewarm) are not reportedProcess.getStartElapsedRealtime()It submits the value once, directly, as a distribution metric (
rn.ios.benchmark.ttfd/rn.android.benchmark.ttfd, in ms) to/api/v1/distribution_points. The tags match the existing benchmark metrics. As a distribution, the median and p90 are computed across runs. Because the measurement doesn't depend on the SDK, it can also be checked against the TTFD the SDK reports in RUM.Arm, then restart. In the Synthetics mobile tests, the app is already running when the first
Open deep linkstep runs. A deep link therefore can't start a measurement from process start. So acoldStartdeep link arms the scenario instead: it saves the run type inUserDefaults/SharedPreferences. On the next launch, the app reads the saved config, clears it, and runs the scenario once. In Synthetics, each sample is:A single test can repeat these steps to collect several samples per run.
Motivation
The benchmark app doesn't measure startup today. This PR lets us track the SDK's cold start overhead (instrumented − baseline) over time, next to the existing CPU, memory and FPS metrics.
Additional Notes
BenchmarkVitals):reportFullyDisplayed,armColdStartandconsumeColdStartConfig, on both platforms. A newFullyDisplayed.swiftis registered inproject.pbxproj, and there's a newFullyDisplayed.kt.Scenario.ColdStart,src/scenario/ColdStart/, the helpers intestUtils.ts, and the arming logic inApp.tsx. Other scenarios are unaffected..benchmarks-ci.ymlis unchanged. CI already builds and uploads the app aslatest, which the Synthetics tests pick up.scenario:coldstart.benchmarks/README.md.Review checklist (to be filled by reviewers)