Skip to content

Severe battery drain from gvproxy CPU spin #2895

Description

@ShivaKumar0786

User Story

As a developer building an application on top of OpenShell's MicroVM driver,
I want gvproxy to recover automatically instead of spinning at 100%+ CPU indefinitely after a network hiccup,
so that host machines running my application don't suffer severe battery drain and CPU starvation.

Problem Statement

gvproxy — the network proxy OpenShell's MicroVM driver spawns to bridge a sandbox VM's network to the host — can enter a sustained high-CPU state after a connection failure and never recovers on its own. Triggers observed to precede it include host sleep/wake cycles, VPN reconnects, and other network changes that break an in-flight outbound connection.

Because gvproxy is intentionally long-lived (kept alive for a session's full VM lifetime rather than restarted per command), once wedged it holds a CPU core (or several, if the spin runs across threads) pinned indefinitely with no internal mechanism to detect or interrupt the loop. Observed on two independently affected machines: 500–1100%+ CPU utilization and elevated wakeup rates sustained for periods ranging from 4 to 16+ hours before manual intervention.

Suspected trigger: a spin/retry loop in gvproxy's connection-handling path, commonly attributed to outbound UDP connection failures (e.g. NTP on port 123). However, we captured gvproxy's own log from a confirmed wedged instance (v0.8.8, ~16 hours pinned at 1100%+ CPU) and it shows no repeated ECONNREFUSED/connection-reset/NTP errors at all — just a normal startup sequence, then silence for the entire wedged period. Whatever loop it's stuck in does not appear to log anything per iteration at the default log level, which means log inspection alone won't confirm the specific trigger — a debug build or strace/dtrace-level tracing would likely be needed.

Impact / Why This Matters

Severe, unbounded battery drain on any laptop running an application built on OpenShell's MicroVM driver — a laptop can go from full charge to near-empty in under two hours while sitting idle, with fans running continuously. Beyond battery, the pinned CPU starves other processes on the host, including the sandbox's own next command (added latency for tool execution / agent turns in our case).

Current workaround: manually detect the wedge (via ps/powermetrics) and kill -9 the specific gvproxy PID, then let the driver recreate the sandbox. This requires the user to notice and act — there's no automatic recovery today. (We've added an external watchdog on our integration side that detects and force-kills a wedged gvproxy after ~90s of sustained high CPU, but that's a workaround at the integration layer, not a fix to gvproxy's own retry/backoff behavior.)

Acceptance Criteria

  • gvproxy's CPU returns to near-idle (comparable to its own healthy baseline, low single-digit CPU ms/s) within a bounded time after a triggering connection failure, without requiring a manual kill.
  • After a host sleep/wake cycle, VPN reconnect, or dropped outbound connection, gvproxy backs off with increasing delay between retries instead of spinning in a tight loop — CPU% does not stay above ~90% for more than a few seconds.
  • If gvproxy hits an unrecoverable connection failure, it logs a clear, actionable error (or exits) instead of spinning silently and indefinitely.
  • When gvproxy enters a retry/backoff loop after a connection failure, it emits at least one log line per retry (or a periodic summary), so the condition is diagnosable from logs alone instead of requiring live process tracing.

Reproduction Steps

  1. Start an OpenShell MicroVM sandbox session and leave it running.
  2. Trigger a network disruption while the session is active — put the host to sleep and wake it, reconnect a VPN, or switch networks (Wi-Fi ↔ Ethernet ↔ VPN).
  3. Monitor gvproxy on the host:
    ps -eo pid,pcpu,etime,command | grep -i gvproxy
  4. Observe CPU% climb and stay pinned (500%+ observed in our case) instead of returning to baseline once the network recovers.

Environment

  • gvproxy version: v0.8.8 (confirmed via gvproxy.log startup line: msg="gvproxy version v0.8.8")
  • OpenShell vm-runtime bundle version: 0.0.86 (directory name under vm-runtime/) — confirmed this is a separate version number from gvproxy's own upstream version, not an inconsistency
  • OS: macOS (build 25F80 observed via powermetrics), Apple Silicon
  • Hardware: MacBookPro18,1
  • Driver: MicroVM (openshell-driver-vm)
  • VM runtime: libkrun
  • Reproduced independently on 2 separate machines with the identical gvproxy version/path

Logs

Collected from a confirmed wedged instance — the process's start timestamp (`ps -o lstart=`) matched this log's own startup timestamp to the second, confirming this is the exact wedged process:

time="2026-08-21T18:43:37+05:30" level=warning msg="bad log level \"\", falling back to \"info\""
time="2026-08-21T18:43:37+05:30" level=info msg="gvproxy version v0.8.8"
time="2026-08-21T18:43:37+05:30" level=info msg="waiting for clients..."

This is the entire file — no further lines were written during the ~16 hours this instance ran pinned at 1100%+ CPU. The spin produces no log output at the default level.

Process/energy evidence (two independent machines):

Machine A — sustained ~4+ hrs, 1s sample (powermetrics --show-process-energy):
Name     PID    CPU ms/s  User%   Wakeups(Intr)  Energy Impact
gvproxy  30115  5806.17   10.07   3929.19        5130.20

Machine B — sustained ~16 hrs (etime 15:55:07):
ps -eo pid,pcpu,etime,command:
88766  1142.4  15:55:07  .../vm-runtime/0.0.86/gvproxy -listen-vfkit unixgram:///tmp/osd-gv/791ea62451cc.v -ssh-port -1

powermetrics --show-process-energy:
gvproxy  88766  11161.12  5.41  18217.91  2.90  18220.81  0.00  6698.68

Same-machine healthy baseline, same tools, non-wedged instance of the same process:
gvproxy  <pid>  4.69      —     86.22          0.22

Metadata

Metadata

Assignees

No one assigned

    Labels

    state:triage-neededOpened without agent diagnostics and needs triage

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions