fix(lambda): add post-deregister busy check to prevent terminating active runners

[JVenberg]

Summary

Fixes #5085

The scale-down lambda can terminate an EC2 instance while it's actively running a job. This happens because a job can be assigned to a runner between checking its busy state and calling TerminateInstances.

We hit this in production: a Helm deploy was killed mid-execution when the scale-down lambda terminated the instance 13 seconds after the job started. CloudTrail confirmed the TerminateInstances call came from the scale-down lambda at the exact moment the runner received a shutdown signal.

The race condition

Current flow in removeRunner:

1. Check GitHub API: "Is this runner busy?" -> false
2. A job gets assigned to the runner here
3. Deregister the runner from GitHub
4. Terminate the EC2 instance
5. The in-flight job is killed

The fix

Add a post-deregistration busy re-check:

1. Check busy (fast-path to skip obviously busy runners)
2. Deregister from GitHub (prevents new job assignment server-side)
3. Re-check busy (now stable, since no new jobs can be assigned after deregistration)

If the re-check finds the runner busy, we skip termination and let the instance be cleaned up as an orphan once the job finishes.

Why this is safe

Deregistering a runner does not affect in-flight jobs. The runner worker uses job-scoped OAuth credentials from the job message, not the runner registration:

• JobRunner.cs lines 80-95: the worker creates its own VssConnection using systemConnection credentials from the job message
• The worker never checks runner registration status during execution
• Deregistration only affects the listener (no new job pickup), not the worker (current job)

Test plan

• All 130 existing tests pass
• New test: runner that becomes busy between deregister and re-check is NOT terminated
• New test: runner that returns 404 on post-deregister busy check IS terminated (runner fully removed from GitHub)

[github-actions]

This pull request has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed if no further activity occurs. Thank you for your contributions.

[npwolf]

We independently hit the underlying race condition in production today and posted repro details on the linked issue (#5085 comment). This fix looks correct to us — the reorder (deregister → recheck busy) closes the TOCTOU window since in-flight jobs use job-scoped credentials that don't depend on registration state.

This has been open since March without review — @edersonbrilhante @Brend-Smits @npalm would one of you be able to take a look? Happy to help test if useful.

[edersonbrilhante]

@npwolf Can you fix the conflicts?