[peterschmidt85]

In-place SSH key updates

SSH fleets now support updating ssh_config.identity_file and ssh_config.proxy_jump.identity_file with dstack apply, including per-host overrides. Previously, these changes required re-creating the fleet, which is blocked while runs are active.

Runs continue without reprovisioning hosts or restarting containers. All established SSH connections stay open, including server management connections, attach sessions, and service tunnels. Existing sessions are not forced to reconnect or reauthenticate.

Example

Authorize the replacement public key on the relevant host, update the identity-file path, and apply the fleet. For example, when replacing both keys:

type: fleet
name: rotation

ssh_config:
  user: ubuntu
  identity_file: ~/.ssh/host-new
  hosts:
    - 10.0.0.10
  proxy_jump:
    hostname: jump.example.com
    user: ubuntu
    identity_file: ~/.ssh/jump-new

Then apply the configuration:

$ dstack apply -f fleet.dstack.yml

Found fleet rotation. Detected changes that can be updated in-place:
- Configuration properties:
  - ssh_config.identity_file
  - ssh_config.proxy_jump.identity_file

Per-host changes appear as ssh_config.hosts[0].identity_file or ssh_config.hosts[0].proxy_jump.identity_file in the apply plan. Changes to SSH users, ports, or proxy_jump addresses still require re-creating the fleet.

Removing the old key

For proxy_jump.identity_file, allow about 20 seconds after apply before removing the old public key from the host configured by proxy_jump. Gateways and the optional SSH proxy need time to receive the new key. Removing the old public key does not close sessions already authenticated with it.

Implementation details

Existing SSH authentication

These rules are unchanged by this PR:

• ssh_config.identity_file: the server uses this key to provision fleet hosts and authorize the project public key.
• Project SSH keys: the server uses these for host management; the SSH proxy, in-server proxy, and gateways use them to access hosts and run containers.
• ssh_config.proxy_jump.identity_file: these components use this key for the additional hop through the fleet's proxy_jump, when configured.
• Gateway SSH keys: the server uses each gateway's own key to connect to that gateway, separately from its service tunnels to fleet hosts.

Applying key changes

The server now supports in-place replacement of identity_file and proxy_jump.identity_file, including per-host overrides. Apply waits for affected instances' ongoing processing to finish, then saves the replacement keys in the fleet configuration and those instances' SSH connection details. New server connections through proxy_jump use the replacement key after apply.

The update preserves all established SSH connections. Each component uses the replacement proxy_jump key for new connections and reconnects after receiving it; sessions authenticated with the old key are not disconnected.

Propagating proxy_jump keys

The shared service connection pool now supports replacing a proxy_jump key while retaining the established SSH tunnel, HTTP client, and Unix socket.

• In-server proxy: Reads the current key on the next service request and updates the pooled connection's credentials.
• Gateways: During synchronization, the server compares each service replica's saved proxy_jump key with the instance's current key and sends replacements where needed. Gateways update existing pooled connections' credentials without unregistering replicas or reloading Nginx. If a replica has no pooled connection, they save its new key without opening SSH. They report replacement keys only after saving them, so failed updates are retried during later synchronization.
• SSH proxy: Its code is unchanged. For new attaches, it fetches the current connection details from the server after its existing 10-second cache expires.

Apply does not wait for gateways to receive the new key or for the SSH proxy's cache to expire.

Performance

• In-server proxy: Refreshing the pooled key adds no database queries.
• Gateways: Replacements are batched into one state-file write per gateway replica. Synchronization loads key data once per instance; public-key derivation is cached and runs off the event loop.
• Server: Waiting for instance processing does not hold a database connection.

Closes #4450

Backward compatibility

• Fleet YAML and public client/server API fields are unchanged. In-place key updates require the updated server; older servers still require re-creating the fleet for these changes.
• Propagating proxy_jump changes to existing gateway service replicas requires the updated gateway app, deployed through the existing server-managed gateway update process. The SSH proxy requires no update.

[jvstme]

I'm not sure about updating all replicas' keys in a single request.

• I assume the request may take a long time, since replicas' SSH tunnels are reopened sequentially.
  • This makes the request fragile due to timeouts and network interruptions.
  • There is less visibility on the server — the server can't tell what the gateway is doing and whether it's doing anything, can't set stricter request timeouts.
• The method only handles proxy jump key updates. If we ever decide to update other replica fields, we'll have to introduce more API methods, potentially with a lot of code duplication.

I would look into introducing generic per-replica /{project_name}/services/{run_name}/replicas/{job_id}/update instead

[peterschmidt85]

Existing tunnels aren't reopened here; the update replaces the key file used for future connections. The batch avoids rewriting the full gateway state once per replica.

There is a fallback in get_or_add that can open SSH if the connection is missing from the pool. That should be kept out of this update path.

Making the update API generic sounds reasonable. I'd keep the batching, though: with the current storage, 1,000 individual updates would rewrite the whole gateway state 1,000 times.

[jvstme]

There is a fallback in get_or_add that can open SSH if the connection is missing from the pool. That should be kept out of this update path

Btw, if tunnels are not being reopened, what's the point of using get_or_add there?

[jvstme]

I'd keep the batching, though

Ok

[peterschmidt85]

Fixed.

Btw, if tunnels are not being reopened, what's the point of using get_or_add there?

I reused get_or_add for its key-update logic. It now updates the existing connection directly. If there's no pooled connection, we save the new key without opening SSH.