test(serve): stop unit tests resolving IAM roles over the network

[mohamedzeidan2021]

Problem

ModelBuilder.__post_init__ resolves a default execution role when none is supplied, and resolve_and_validate_role does that with sts:GetCallerIdentity followed by a paginated iam:SimulatePrincipalPolicy. So every unit test that builds a ModelBuilder without role_arn makes live IAM calls — which succeed only because CI credentials happen to be present.

IAM throttles on an account-wide quota, so those builds fail with:

botocore.exceptions.ClientError: An error occurred (Throttling) when calling the
SimulatePrincipalPolicy operation (reached max retries: 4): Rate exceeded

…on whichever test happened to be running when the limit hit.

Scale, from production

Over all 531 builds of sagemaker-python-sdk-ci-health-unit-test-v3 with SUBMODULE=sagemaker-serve (CloudWatch logs retrieved for 123 of 123 failures):

Window since 2026-09-23
Decided builds	184 (142 pass / 42 fail)
Observed pass rate	77.2%
Throttling share of failures	37 of 42 (88%)
Pass rate with throttling removed	97.3%
Residual failures	5, all public-PyPI download flake
Genuine test failures	0

100% of throttling in that window — 37 of 37 builds, 90 of 90 per-env instances — comes from tests/unit/test_model_builder_methods.py, the one remaining file whose tests construct ModelBuilder without a role and without stubbing the resolver. All 23 of its unprotected tests have thrown it in production. Cost so far: roughly 62 build-hours, each needing a developer re-run.

The failures are not reproducible. Each build runs the suite three times (py310/py311/py312). Throttling failures hit only one of the three 89% of the time; genuine failures hit all three 100% of the time. Independently, of 36 commits built more than once, 22 produced both a pass and a fail — 17 of those were throttling, and none were genuine.

Why a conftest

This generalises the fixture #6272 added to test_model_builder_coverage_boost.py, which drove that file from 83% of throttling occurrences to zero. test_bedrock_model_builder.py and test_ai_inference_recommender/test_model_builder_recommendations.py carry the same fixture and have never thrown a throttle in 531 builds — a clean control for the mechanism. Putting it in tests/unit/conftest.py covers the package instead of one file at a time.

Two details:

• The stub passes the provided role through rather than returning a fixed value, matching the real contract. Four of the five call sites pass a real role, and tests assert on what comes back.
• resolve_and_validate_role is patched once per importing module. model_builder.py:63 and bedrock_model_builder.py:39 each bind it with a module-level from ... import, so each holds its own reference — patching the definition site in sagemaker.core.helper.iam_role_resolver would rebind neither.

Tests that patch the resolver themselves still win, since their patch is applied inside the fixture's and rebinds the attribute last.

Verification

Measured with deliberately invalid credentials and a transport-level request recorder with a positive control, so a silent failure to record cannot be mistaken for absence of traffic (it reports 23 requests from 23 tests without this change, 0 with it).

	Before	After
serve unit suite	67 failed, 2087 passed	44 failed, 2110 passed
Newly broken tests	—	0
Tests issuing STS calls	24	1
STS/IAM errors in the run	present	0

The single remaining STS caller is the telemetry daemon thread, attributed outside any test and unrelated to this change.

The 44 remaining failures are all s3:GetObject against the JumpStart cache. Those succeed in CI and cause no CI failures — they only fail locally because invalid credentials break S3 before IAM is reached. Left for a separate change.

Coverage held whole

• serve/model_builder.py gains 5 covered lines, because 23 tests now run to completion instead of aborting on an AWS error.
• 18 lines of sagemaker-core/helper/iam_role_resolver.py stop being covered by serve — they were reached only by making real IAM calls. 17 were already covered by that module's own tests.
• The 18th (the early return None for role types with no config path, which serve was exercising incidentally via role_type="serving") is now covered deliberately by the new TestConfigDefaultRole in sagemaker-core/tests/unit/helper/test_iam_role_resolver.py. _resolve_config_default_role had no direct tests before.

Scope

Test files only — no production code changes.

Note for reviewers

The deeper defect is that constructing a ModelBuilder performs a paginated IAM call at all. This change stops the bleeding; making role resolution lazy would be the cure, and is worth considering separately.

[github-actions]

🤖 AI code review

This PR adds a package-level conftest.py for sagemaker-serve unit tests that stubs resolve_and_validate_role at both module import sites (model_builder and bedrock_model_builder) via an autouse fixture, keeping IAM/STS calls off the network to eliminate SimulatePrincipalPolicy throttling flakiness. It also adds unit tests for the new _config_path_for_role_type / _resolve_config_default_role helpers in iam_role_resolver. The change is test-only, well-reasoned, and consistent with prior precedent (PR #6272). I found no correctness issues: all serve call sites invoke the resolver with keyword arguments that the stub's **kwargs safely absorbs, the config-default short-circuit test patches the correct definition site, and the autouse-fixture ordering claim holds. No blocking findings.

_{Reviewed commit 120badd. Automated review; verify before acting.}