test: add ShellSpec coverage for init-aks-custom-cloud.sh by Devinwong · Pull Request #8886 · Azure/AgentBaker

Devinwong · 2026-07-10T19:17:01Z

[No logic in init-aks-custom-cloud.sh is changed. Just make it testable]

Resolves conflicts from PR #8692 after #8096 consolidated the four separate init-aks-custom-cloud scripts into one.

Related: ADO#36479419

Changes

Script restructuring (init-aks-custom-cloud.sh) — testability only

Moved all function definitions above a $("{SOURCED:+return 0}") guard (following localdns.sh pattern)
Made repo-depot helpers use env-var-overridable paths (APT_SOURCES_LIST_D_DIR, APT_KEYRINGS_DIR, YUM_REPOS_DIR, etc.)
Extracted determine_cert_endpoint_mode() from inline case statement
Removed misleading distribution is log (variable was never set in consolidated script)

Test Coverage Summary

Wiring tests (grep-based) — ca-refresh mode structural invariants

Verifies the script contains correct patterns for: action parsing (init/ca-refresh), location fallback from LOCATION, case-insensitive normalization, ussec/usnat to legacy mapping, refresh schedule gating, early exit in ca-refresh mode, and LOCATION passthrough to cron/systemd commands.

Functional tests — sourced and executed

Function	Cloud endpoints tested	Key assertions
determine_cert_endpoint_mode	ussec, usnat, USSecWest, USNatEast, usgovvirginia (fairfax), chinaeast2 (mooncake), francesouth (bleu), empty	ussec/usnat to legacy; all others to rcv1p
init_ubuntu_main_repo_depot	bleu, fairfax, mooncake, ussec	Correct URIs in ubuntu.sources; no archive.ubuntu.com/security.ubuntu.com/packages.microsoft.com leaks; existing sources backed up and removed
init_ubuntu_pmc_repo_depot	bleu	microsoft-prod.sources + microsoft-prod-testing.sources created; URIs point at depot; GPG keys installed; no packages.microsoft.com leak
check_url	bleu	Returns success on 200; stdout includes "Checking url"
init_mariner_repo_depot	ussec	extended/nvidia/cloud-native repos created from extras template; all baseurl rewritten to repodepot; no original packages.microsoft.com URLs remain
init_azurelinux_repo_depot	usnat, mooncake	All 7 repo files created (amd, base, cloud-native, extended, ms-non-oss, ms-oss, nvidia); baseurl points at depot; no packages.microsoft.com

Coverage: Ubuntu, Mariner, Azure Linux | All 5 custom clouds (bleu, fairfax, mooncake, ussec, usnat)

Supersedes

test: add ShellSpec coverage for init-aks-custom-cloud*.sh #8692 (original PR, now conflicted after feat(rcv1p): unify cert bootstrap flow and add Windows CA refresh task #8096 merged)

Resolves conflicts from PR #8692 after #8096 consolidated the four separate init-aks-custom-cloud scripts into one. Changes: - Restructure init-aks-custom-cloud.sh for source-safety: move all function definitions above a guard (following the localdns.sh pattern) so ShellSpec can Include the file and exercise functions directly. - Make repo-depot helper functions use env-var-overridable paths (APT_SOURCES_LIST_D_DIR, APT_KEYRINGS_DIR, etc.) for testability. - Remove misleading 'distribution is' log line. - Add __SOURCED__ guard for set -x to suppress trace noise in tests. - Merge spec file: keep grep-based ca-refresh wiring tests (from #8096) and add functional tests for init_ubuntu_main_repo_depot, init_ubuntu_pmc_repo_depot, and check_url. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

This PR makes parts/linux/cloud-init/artifacts/init-aks-custom-cloud.sh safe to source (so ShellSpec can Include it) and adds ShellSpec coverage for key repo-depot and URL-checking helpers used in custom-cloud node boot.

Changes:

Refactors init-aks-custom-cloud.sh to be source-safe by moving execution-only logic below an __SOURCED__ guard and gating set -x when sourced.
Makes several paths/env inputs overridable (e.g., apt sources/keyrings locations, wireserver endpoint) to enable hermetic tests.
Adds/expands ShellSpec tests: keeps grep-based “wiring” checks and adds functional tests for init_ubuntu_main_repo_depot, init_ubuntu_pmc_repo_depot, and check_url.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
spec/parts/linux/cloud-init/artifacts/init_aks_custom_cloud_spec.sh	Adds wiring + functional ShellSpec coverage for repo-depot helpers and URL checking.
parts/linux/cloud-init/artifacts/init-aks-custom-cloud.sh	Refactors for source-safety and testability (guarded execution, env-overridable paths/endpoints).

…ngs_dir Address review comment: variables in add_key_ubuntu are now declared local to prevent state leakage, arguments are properly quoted, and the unused keyrings_dir variable is removed (derive_key_paths has its own). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

Add tests covering different custom cloud scenarios per ADO#36479419: - determine_cert_endpoint_mode: parameterized tests for ussec, usnat (legacy), fairfax/mooncake/bleu (rcv1p), and empty location. - init_mariner_repo_depot: verifies repo creation and URL rewriting with a ussec-like endpoint. - init_azurelinux_repo_depot: verifies repo file generation for usnat and mooncake endpoints. - init_ubuntu_main_repo_depot: additional tests with fairfax, mooncake, and ussec endpoints. Production script changes (testability only): - Extract determine_cert_endpoint_mode() from inline case statement. - Make init_mariner_repo_depot and init_azurelinux_repo_depot use overridable YUM_REPOS_DIR (defaults to /etc/yum.repos.d). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

rchincha · 2026-07-14T22:38:34Z

+# Determines the certificate endpoint mode based on location.
+# Returns "legacy" for ussec/usnat regions, "rcv1p" for all others.
+# Usage: cert_endpoint_mode=$(determine_cert_endpoint_mode "$location")
+function determine_cert_endpoint_mode {


The extracted determine_cert_endpoint_mode() function is the direct behavioral surface of RCV1P's cloud selection. It's now tested for 8 inputs (ussec, usnat, USSecWest, USNatEast, usgovvirginia, chinaeast2, francesouth, empty). However, there's no test that verifies the refresh_location env-var/arg2 handoff into it actually happens at the script top-level (grep-only). If someone in a future refactor accidentally passes $1 instead of $refresh_location to determine_cert_endpoint_mode , all unit tests still pass. Suggest adding a script-level ShellSpec that runs LOCATION=usseceast bash init-aks-custom-cloud.sh ca-refresh and asserts the emitted AKS.CSE.rcv1p.certEndpointMode event says mode=legacy . This locks the whole RCV1P mode-selection wiring, not just the isolated function.

Not sure if this is in scope of this PR though.

This is a valid concern. But it seems it will need more changes. Let me follow up in a separate PR.

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

init_ubuntu_main_repo_depot templates the repodepot URL verbatim with no per-cloud branching, so the fairfax/mooncake/ussec cases exercised the same path as the generic placeholder test. Remove the redundant block; the generic repodepot.example.com test already covers the pass-through. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

Neutralize enumerated cloud regions (ussec/usnat) to repodepot.example.com and collapse the redundant Azure Linux Mooncake case, matching the init_ubuntu_main_repo_depot cleanup. These functions template the repodepot URL verbatim with no per-cloud branching, so one placeholder test covers it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

- Harden WIRESERVER_ENDPOINT: only honor env override when sourced for tests - Create parent dirs for env-overridable paths (apt sources.list.d, yum.repos.d, OpenSSL cert target) so *_DIR test overrides are robust; no-op on prod defaults - OpenSSL bundle copy: warn on missing source, skip on same-file, no false success - Assert all 7 Azure Linux repo files in spec - Document the __SOURCED__ convention at top of file - Add wiring test pinning the determine_cert_endpoint_mode refresh_location handoff Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

Set __SOURCED__=1 before Include so ShellSpec loads only the function definitions. The script's ${__SOURCED__:+return 0} guard is a no-op when __SOURCED__ is unset, so without this the Include could fall through into the top-level provisioning path (wireserver calls, cron install, exit) and cause side effects. Matches the sourcing convention documented in the script header and used in cse_main_spec.sh. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

… failure path Co-authored-by: Devinwong <13621569+Devinwong@users.noreply.github.com>

aks-node-assistant · 2026-07-15T16:53:06Z

AgentBaker Linux gate detective RCA for failed run: https://msazure.visualstudio.com/CloudNativeCompute/_build/results?buildId=172266842

Failed job/task: Run AgentBaker E2E / Run AgentBaker E2E.
L1 surface failure: gotestsum failed after broad near-immediate E2E skips/failures; first sampled signature is $(System.Collections.Hashtable.sample); 24 missing-image metadata lines were observed.
L2 corroboration: ADO timeline marks only the E2E job/task failed; test results show E2E failures; the run log reports current-build image lookup missing before VM provisioning.
L3 root-cause challenge: Product/CSE/VHD regression is less likely because this stops at image discovery before node creation and recurs across unrelated PRs. Best match is known wiki signature �2e-vhd-build-id-unresolved-macro-missing-images (pipeline/config image metadata issue), tracked by repair item #38812052.
Confidence: medium-high. Strongest alternative: PR-specific change altered expected images, but recurrence across PRs makes gate infrastructure/config more likely.
Next action: treat as gate flake; follow repair item #38812052 and the wiki row: https://dev.azure.com/msazure/09706533-03bf-4b43-9a9b-b49c75429646/_wiki/wikis/ed4a85e9-1085-4151-a39b-2753523eba2b?pagePath=%2FAKS%2FSIGs%20and%20Teams%2FAKS%20Components%2FSIG%3A%20Node%20Lifecycle%2FAI%20Agent%20Knowledge%2FAgentBaker%20Gate%20PR%20Pipeline%20Flakiness%2FItems%20With%20Repair%20Items

Copilot AI review requested due to automatic review settings July 10, 2026 19:17

Devinwong requested review from AbelHu, SriHarsha001, awesomenix, calvin197, cameronmeissner, djsly, ganeshkumarashok, karenychen, lilypan26, mxj220, pdamianov-dev, phealy, r2k1, runzhen, sulixu, surajssd, timmy-wright, titilambert, xuexu6666 and zachary-bailey as code owners July 10, 2026 19:17

Copilot started reviewing on behalf of Devinwong July 10, 2026 19:17 View session

Copilot AI reviewed Jul 10, 2026

View reviewed changes

Comment thread parts/linux/cloud-init/artifacts/init-aks-custom-cloud.sh Outdated

Copilot AI review requested due to automatic review settings July 10, 2026 20:06

Copilot started reviewing on behalf of Devinwong July 10, 2026 20:07 View session

Copilot AI reviewed Jul 10, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings July 10, 2026 20:22

Copilot AI reviewed Jul 14, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings July 14, 2026 22:34

Copilot started reviewing on behalf of Devinwong July 14, 2026 22:35 View session

rchincha reviewed Jul 14, 2026

View reviewed changes

Copilot AI reviewed Jul 14, 2026

View reviewed changes

Comment thread parts/linux/cloud-init/artifacts/init-aks-custom-cloud.sh Outdated

rchincha reviewed Jul 14, 2026

View reviewed changes

Comment thread parts/linux/cloud-init/artifacts/init-aks-custom-cloud.sh

rchincha reviewed Jul 14, 2026

View reviewed changes

Comment thread parts/linux/cloud-init/artifacts/init-aks-custom-cloud.sh

Copilot AI review requested due to automatic review settings July 14, 2026 22:47

Copilot started reviewing on behalf of Devinwong July 14, 2026 22:47 View session

Copilot AI reviewed Jul 14, 2026

View reviewed changes

Comment thread parts/linux/cloud-init/artifacts/init-aks-custom-cloud.sh

Comment thread spec/parts/linux/cloud-init/artifacts/init_aks_custom_cloud_spec.sh Outdated

Comment thread spec/parts/linux/cloud-init/artifacts/init_aks_custom_cloud_spec.sh Outdated

Copilot AI review requested due to automatic review settings July 14, 2026 22:51

Copilot started reviewing on behalf of Devinwong July 14, 2026 22:52 View session

Copilot AI reviewed Jul 14, 2026

View reviewed changes

Comment thread parts/linux/cloud-init/artifacts/init-aks-custom-cloud.sh

Comment thread parts/linux/cloud-init/artifacts/init-aks-custom-cloud.sh Outdated

Comment thread parts/linux/cloud-init/artifacts/init-aks-custom-cloud.sh

Copilot AI review requested due to automatic review settings July 15, 2026 00:18

Copilot started reviewing on behalf of Devinwong July 15, 2026 00:19 View session

Copilot AI reviewed Jul 15, 2026

View reviewed changes

Comment thread spec/parts/linux/cloud-init/artifacts/init_aks_custom_cloud_spec.sh

Copilot AI review requested due to automatic review settings July 15, 2026 00:30

Copilot started reviewing on behalf of Devinwong July 15, 2026 00:30 View session

Devinwong mentioned this pull request Jul 15, 2026

test: add script-level ShellSpec to lock RCV1P mode-selection wiring (follow-up to #8886) #8946

Open

4 tasks

Copilot AI reviewed Jul 15, 2026

View reviewed changes

Comment thread parts/linux/cloud-init/artifacts/init-aks-custom-cloud.sh

Copilot started work on behalf of Devinwong July 15, 2026 00:36 View session

fix: downgrade misleading ERROR to Warning (stderr) in CA bundle copy…

9d4b0b6

… failure path Co-authored-by: Devinwong <13621569+Devinwong@users.noreply.github.com>

Copilot AI review requested due to automatic review settings July 15, 2026 00:39

Devinwong removed the request for review from Copilot July 15, 2026 00:39

Copilot finished work on behalf of Devinwong July 15, 2026 00:40

Uh oh!

Conversation

Devinwong commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Script restructuring (init-aks-custom-cloud.sh) — testability only

Test Coverage Summary

Wiring tests (grep-based) — ca-refresh mode structural invariants

Functional tests — sourced and executed

Supersedes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rchincha Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Devinwong Jul 15, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

aks-node-assistant Bot commented Jul 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Devinwong commented Jul 10, 2026 •

edited

Loading

rchincha Jul 14, 2026 •

edited

Loading