kubernetes: clean up temporary provisioning scripts on the management server by nikolauseppinger · Pull Request #13592 · apache/cloudstack

nikolauseppinger · 2026-07-10T14:56:09Z

Description

The CKS action workers copy provisioning scripts to the cluster nodes via
retrieveScriptFiles() + copyScripts() / copyScriptFile(). Each script is
written to a local temp file with File.createTempFile(filename, ".sh") in
retrieveScriptFile(), but that local copy on the management server is never
deleted after it has been SCP'd to the node.

As a result, every cluster deploy, provider/CSI deploy, autoscale enable, PV
cleanup and Kubernetes upgrade leaks one *.sh file per script into the
management server's temp directory, growing without bound over the cluster
lifetime:

-rw-r--r--. 1 cloud cloud 3230 ... autoscale-kube-cluster10402639792822688389.sh
-rw-r--r--. 1 cloud cloud 7197 ... delete-pv-reclaimpolicy-delete10664788910304997694.sh
-rw-r--r--. 1 cloud cloud 2303 ... deploy-cloudstack-secret10522058804386657778.sh
-rw-r--r--. 1 cloud cloud 1811 ... deploy-csi-driver1118819744193036791.sh
-rw-r--r--. 1 cloud cloud 1472 ... deploy-provider10072339396849174814.sh
-rw-r--r--. 1 cloud cloud 4971 ... upgrade-kubernetes.sh10982436999748062804.sh
... (dozens more, one set per CKS action)

This PR adds cleanupScriptFiles() / deleteScriptFileQuietly() to the base
KubernetesClusterActionWorker and invokes it in a finally block around
every copy site:

KubernetesClusterActionWorker.deployProvider()
KubernetesClusterActionWorker.deployCsiDriver()
KubernetesClusterResourceModifierActionWorker.autoscaleCluster()
KubernetesClusterResourceModifierActionWorker.deletePVsWithReclaimPolicyDelete()
KubernetesClusterUpgradeWorker (overrides cleanupScriptFiles() to also remove
its upgrade-kubernetes.sh temp file, cleaned up after the per-node upgrade loop
completes since the same file is reused across nodes)

There is no functional change to the provisioning itself: the scripts are still
written, copied and executed exactly as before; only the leftover local copies
on the management server are removed once they have been transferred.

Types of changes

Bug fix (non-breaking change which fixes an issue)

Bug Severity

Minor

Observed on a CloudStack 4.22.1 management server running CKS clusters; the code
path is unchanged on main.

… server The CKS action workers copy provisioning scripts (deploy-cloudstack-secret, deploy-provider, deploy-csi-driver, delete-pv-reclaimpolicy-delete, autoscale-kube-cluster and upgrade-kubernetes.sh) to the cluster nodes via retrieveScriptFiles() + copyScripts()/copyScriptFile(). Each script is written to a local temp file via File.createTempFile(), but that local copy is never deleted after it has been SCP'd to the node. As a result every cluster deploy, provider/CSI deploy, autoscale enable, PV cleanup and Kubernetes upgrade leaks one *.sh file per script into the management server's temp directory, growing without bound over the cluster lifetime. On a long-running management server this accumulates to hundreds of stale scripts in /tmp. This adds cleanupScriptFiles()/deleteScriptFileQuietly() to the base action worker and invokes it in a finally block around every copy site (deployProvider, deployCsiDriver, autoscaleCluster, deletePVsWithReclaimPolicyDelete and the upgrade worker), so the local temp copies are removed once they have been transferred to the node. The upgrade worker additionally removes its upgrade-kubernetes.sh temp file after the node loop completes. No functional change to the provisioning itself: the scripts are still written, copied and executed exactly as before; only the leftover local copies are deleted. Covered by new unit tests for cleanupScriptFiles()/deleteScriptFileQuietly() (temp files deleted, null/missing files handled without throwing).

codecov · 2026-07-14T06:08:17Z

Codecov Report

❌ Patch coverage is 23.07692% with 30 lines in your changes missing coverage. Please review.
✅ Project coverage is 19.47%. Comparing base (a503a52) to head (9f72e75).
⚠️ Report is 20 commits behind head on main.

Files with missing lines	Patch %	Lines
...KubernetesClusterResourceModifierActionWorker.java	0.00%	15 Missing ⚠️
...r/actionworkers/KubernetesClusterActionWorker.java	52.94%	7 Missing and 1 partial ⚠️
.../actionworkers/KubernetesClusterUpgradeWorker.java	0.00%	7 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff            @@
##               main   #13592   +/-   ##
=========================================
  Coverage     19.47%   19.47%           
- Complexity    19351    19355    +4     
=========================================
  Files          6296     6296           
  Lines        568368   568394   +26     
  Branches      69622    69622           
=========================================
+ Hits         110673   110717   +44     
+ Misses       445618   445597   -21     
- Partials      12077    12080    +3

Flag	Coverage Δ
uitests	`3.44% <ø> (ø)`
unittests	`20.74% <23.07%> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

shwstppr · 2026-07-16T06:43:24Z

@blueorangutan package

blueorangutan · 2026-07-16T06:44:05Z

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2026-07-16T07:30:22Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 18588

DaanHoogland · 2026-07-20T11:45:34Z

@blueorangutan test

blueorangutan · 2026-07-20T11:46:04Z

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

Damans227 · 2026-07-20T13:51:55Z

    }

+    @Override
+    protected void cleanupScriptFiles() {


Doesn't this delete upgradeScriptFile too early? deployProvider() runs per node in the loop and calls cleanupScriptFiles() in its own finally, and since this is an UpgradeWorker, that hits this override and deletes upgradeScriptFile before runInstallScriptOnVM() (next line) SCPs it. Could this break the upgrade the first time a node needs the provider re-deployed?

Damans227 · 2026-07-20T13:52:30Z

+        deleteScriptFileQuietly(autoscaleScriptFile);
+    }
+
+    protected void deleteScriptFileQuietly(File file) {


Should the field be set to null after deleting? Right now it still points to a file that no longer exists. If it's used again before retrieveScriptFiles() runs, it could silently try to work with a deleted file.

Damans227 · 2026-07-20T13:53:46Z

                logMessage(Level.INFO, "Provider files missing. Adding them now", null);
                retrieveScriptFiles();
-                copyScripts(publicIpAddress, sshPort);
+                try {


This same try { copyScripts(...) } finally { cleanupScriptFiles(); } block shows up 3 times (here, deployCsiDriver(), and autoscaleCluster()). Could this be a small shared helper, like copyScriptsWithCleanup(nodeAddress, sshPort), to avoid repeating it?

Damans227 · 2026-07-20T13:54:21Z

+     * once they have been copied to the cluster node(s). Without this, every deploy/autoscale/PV-cleanup
+     * action leaks a *.sh file in the management server's tmp directory.
+     */
+    protected void cleanupScriptFiles() {


Nit: could this loop over a list of the 5 fields instead of calling deleteScriptFileQuietly 5 times by hand? Would be one line instead of five, and less to update if a new script field gets added later.

Damans227 · 2026-07-20T13:56:01Z

+
+    @Test
+    public void testCleanupScriptFilesDeletesAllTempFiles() throws Exception {
+        actionWorker.deploySecretsScriptFile = File.createTempFile("deploy-cloudstack-secret", ".sh");


Nit: these strings duplicate actionWorker.deploySecretsScriptFilename and friends. Could reuse those constants instead?

boring-cyborg Bot added the component:kubernetes label Jul 10, 2026

DaanHoogland requested a review from Damans227 July 20, 2026 11:45

DaanHoogland added the status:needs-testing label Jul 20, 2026

DaanHoogland added this to the 4.24.0 milestone Jul 20, 2026

DaanHoogland added the status:needs-review label Jul 20, 2026

Damans227 reviewed Jul 20, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

kubernetes: clean up temporary provisioning scripts on the management server#13592

kubernetes: clean up temporary provisioning scripts on the management server#13592
nikolauseppinger wants to merge 1 commit into
apache:mainfrom
nikolauseppinger:kubernetes/cleanup-temp-provisioning-scripts

nikolauseppinger commented Jul 10, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Jul 14, 2026 •

edited

Loading

Uh oh!

shwstppr commented Jul 16, 2026

Uh oh!

blueorangutan commented Jul 16, 2026

Uh oh!

blueorangutan commented Jul 16, 2026

Uh oh!

DaanHoogland commented Jul 20, 2026

Uh oh!

blueorangutan commented Jul 20, 2026

Uh oh!

Damans227 Jul 20, 2026

Uh oh!

Damans227 Jul 20, 2026

Uh oh!

Damans227 Jul 20, 2026

Uh oh!

Damans227 Jul 20, 2026

Uh oh!

Damans227 Jul 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

nikolauseppinger commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Bug Severity

Uh oh!

codecov Bot commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

shwstppr commented Jul 16, 2026

Uh oh!

blueorangutan commented Jul 16, 2026

Uh oh!

blueorangutan commented Jul 16, 2026

Uh oh!

DaanHoogland commented Jul 20, 2026

Uh oh!

blueorangutan commented Jul 20, 2026

Uh oh!

Damans227 Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

Damans227 Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

Damans227 Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

Damans227 Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

Damans227 Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

nikolauseppinger commented Jul 10, 2026 •

edited

Loading

codecov Bot commented Jul 14, 2026 •

edited

Loading