test: rotate Terraform E2E identities to reduce RBAC flakiness

[Mani Bindra (maniSbindra)]

Summary

Alternate the Linux Terraform E2E tests between two service principals to reduce stale Azure RBAC state carried between consecutive tests.

This change also:

• adds targeted provisioning for the alternate Terraform service principal;
• wires and validates the alternate credentials in the E2E workflow;
• preserves the existing single gotestsum process, test order, reporting, and failure aggregation;
• runs the Linux Terraform CLI test first with the primary identity;
• reduces the post-detachment propagation delay from 45 seconds to 1 second.

Motivation

The Terraform E2E suite currently reuses one service principal across all tests. Azure role-assignment removal is eventually consistent, so permissions from one test can remain effective when the next test begins. This can cause permission discovery to depend on residual RBAC state and produce intermittent failures.

Using alternating identities gives each service principal additional recovery time while the other identity runs the next test. It also avoids applying a long fixed delay to every MPF invocation.

This work is related to #231, but does not close it. Azure RBAC propagation remains eventually consistent, and additional stabilization may still be needed for tests with exact permission assertions.

Implementation

• Add mpf-terraform-linux-alt-e2e-sp as an alternate Linux Terraform identity.
• Select primary and alternate credentials using a concurrency-safe sequence in the Go E2E harness.
• Fall back to the primary identity for local runs when alternate credentials are entirely absent.
• Fail early when alternate credentials are only partially configured.
• Add workflow preflight checks for both Terraform identities.
• Add targeted --target terraform_linux_alt provisioning support to the Bash and PowerShell scripts.
• Prevent targeted provisioning from overwriting an existing credential output file.
• Keep ARM, Bicep, and Windows Terraform behavior unchanged.
• Retain the existing five-second waits after the initial role assignment and after each custom-role update.

E2E results

The branch completed successfully with the original 15-second experiment and with the reduced one-second delay.

With the one-second delay:

• five full E2E workflow runs succeeded;
• the latest four spaced runs all completed successfully across every matrix job;
• one closely timed follow-up run failed because TestTerraformAuthorizationPermissionMismatch discovered the intermittent additional Microsoft.Resources/subscriptions/providers/read permission.

The isolated failure indicates that identity rotation reduces adjacent-test contamination but cannot completely eliminate Azure RBAC propagation variability across closely spaced workflow runs.

Related issue

References #231.

[Copilot]

Pull request overview

This PR reduces Linux Terraform E2E flakiness caused by stale Azure RBAC state by alternating Terraform E2E tests between a primary and an alternate service principal, and wiring the alternate credentials through provisioning scripts and the GitHub Actions workflow.

Changes:

• Add Terraform E2E identity selection logic (primary/alternate) and update Terraform E2E tests to use it.
• Extend SP provisioning scripts to support a terraform_linux_alt target and targeted provisioning safeguards.
• Update the E2E workflow to preflight-check both Terraform identities and pass alternate credentials into Linux Terraform E2E runs.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
Taskfile.yml	Adds a focused unit test run for the identity selector in the standard test task.
scripts/create-e2e-service-principals.sh	Adds --target support, introduces terraform_linux_alt, and prevents targeted runs from overwriting an existing output file.
scripts/create-e2e-service-principals.ps1	Mirrors targeted provisioning and terraform_linux_alt support for PowerShell.
pkg/usecase/mpfService.go	Reduces the post-detach RBAC propagation sleep from 45s to 1s.
e2eTests/e2eTerraformWithImportAndTargeting_test.go	Switches Terraform E2E tests to use the Terraform identity selector args helper.
e2eTests/e2eTerraformInvalid_test.go	Switches Terraform invalid-path E2E tests to use the Terraform identity selector args helper.
e2eTests/e2eTerraformIdentity_test.go	Adds identity selection logic + selector unit tests.
e2eTests/e2eTerraformAuthPermissionMismatch_test.go	Switches Terraform E2E test to use the Terraform identity selector args helper.
e2eTests/e2eTerraformAuthorizationRequestDenied_test.go	Switches Terraform E2E test to use the Terraform identity selector args helper.
e2eTests/e2eTerraform_test.go	Switches Terraform E2E tests to use the Terraform identity selector args helper.
.github/workflows/e2e-tests.yaml	Adds Terraform credential preflight checks, runs Linux terraform CLI tests first with the primary identity, and passes alternate creds into terraform E2E runs.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.