Skip to content

fix: stop the RAPI command queue fragmenting the heap - #6

Open
RAR wants to merge 2 commits into
OpenEVSE9from
fix/rapi-queue-heap-fragmentation
Open

fix: stop the RAPI command queue fragmenting the heap#6
RAR wants to merge 2 commits into
OpenEVSE9from
fix/rapi-queue-heap-fragmentation

Conversation

@RAR

@RAR RAR commented Aug 13, 2026

Copy link
Copy Markdown

Problem

On an ESP32 running the OpenEVSE WiFi firmware, the largest contiguous free
heap block falls steadily until large allocations start failing, ending in a
panic reboot. Total free heap barely moves the whole time, so a free_heap
check shows nothing wrong — this is fragmentation, not a leak:

over ~2 days
free_heap holds at 70–85 KB
largest free block 57 KB → 2 KB

Anything needing a large contiguous buffer — TLS, OTA — fails long before the
heap looks full. OTA in particular starts failing partway through the download.

Root cause

The command queue is a static array of CommandItem, each holding an Arduino
String and a std::function. Queue::pop() copies the item out but leaves
the slot populated:

bool pop(T &item) {
  item = values[tail];       // slot still holds a copy
  tail = nextSlot(tail);

So all RAPI_MAX_COMMANDS slots permanently retain a String and a callback on
the heap. Each time a slot is reused, those two blocks are freed and
re-allocated wherever the allocator happens to be — walking steadily further
into the largest free run. They act as a set of slowly migrating pins that
ratchet the largest contiguous block downward.

The callbacks land on the heap because std::function allocates for any
callable above its small-object buffer (8 bytes here), and this library's
callbacks are [this, callback] lambdas at 20 bytes. Every queued command
allocated.

Confirmed by walking the heap block layout with heap_caps_walk(): a few
kilobytes of 20-byte blocks, scattered through what had been a 58 KB free run,
cut it to 31 KB. Symbolizing them (a captured this pointing into the global
OpenEVSE object, plus the manager function pointers) identified them as the
OpenEVSEClass::begin / EvseMonitor::begin completion lambdas.

A natural control: two otherwise identical units, same firmware. The one whose
controller link was down — no sustained RAPI traffic — held its largest block at
55,284 bytes for 15 hours. The one sending 1.3 commands/sec collapsed to 2 KB.

Changes

  • Queue::pop() releases the slot, so a popped item is not kept alive by the
    queue. push() takes a const reference rather than a copy.
  • CommandItem::command becomes a fixed char[]. Commands longer than the
    buffer are rejected with the already-defined RAPI_RESPONSE_CMD_TOO_LONG
    rather than truncated — a truncated $SC 32 is a valid command with a
    different meaning.
  • The completion callback gets fixed inline storage instead of std::function.
    A callable too large to fit is now a compile error, so the failure mode is
    a build break rather than a silent return to heap allocation.

Trades RAPI_MAX_COMMANDS * ~140 bytes of .bss, which cannot fragment, for
the heap blocks that could. No public API change; all existing call sites
compile unmodified.

Second commit: async event guards

Found while auditing the same paths. onEvent() read tokens 1..4 without
checking the token count. getToken() returns NULL at or past the count and
strtol(NULL) dereferences it, so an async frame with fewer fields than
expected panics the host. The checksum is verified before this runs, so this is
not corruption — it is a controller sending a shorter frame than assumed, which
a different firmware version legitimately can. Every branch now checks the count
first, as the request handlers elsewhere in the file already do. $AB requires
three tokens rather than two because the firmware string is handed to the boot
callback and consumers dereference it without a NULL check.

Also in that commit: every public method began with if (!_sender) { return; },
abandoning the caller's callback. A caller that sets a pending flag and waits
for completion waited forever, with no result and no error. Those now complete
with a new RAPI_RESPONSE_NOT_CONNECTED.

Validation

Built and run on an ESP32 (openevse_wifi_tft_v1). RAPI functionally intact —
1,581 commands with a single failure (the usual boot-time timeout), live voltage
and temperature reading back from the controller normally.

Largest free block, same unit, same workload:

before after
at boot 61,428 55,284
after ~5 min 49,140 47,092
after ~22 min / 1.5k commands falling toward 32,756 53,236 (rising)

Before, it only ever fell. After, freed blocks coalesce again and the region
that was being carved up stays clean.

Known hazards not addressed here

Flagging rather than fixing, since both are larger decisions than this change
should make:

  1. _commandComplete() keeps _waitingForReply set across the handler call, so
    a handler that itself calls sendCmdSync()/flush() re-enters loop() and
    runs the same handler a second time — after which _sendNextCmd() reassigns
    _completeHandler while its operator() is still on the stack. No current
    caller does this. Fixing it properly means deciding whether a nested
    synchronous send should be allowed at all.
  2. getToken() returns interior pointers into a buffer the next response
    overwrites, so callbacks must copy anything they keep. Currently undocumented.

A follow-up branch collects several smaller items from the same audit
(sendCmdSync discarding its timeout argument, OPENEVSE_VFLAG_DEFAULT
expanding to undefined ECVF_* names, getCurrentCapacity's declared parameter
names not matching the order actually passed). Kept separate so this change
stays reviewable on its own.

RAR added 2 commits August 13, 2026 09:21
The command queue is a static array of CommandItem, each holding an Arduino
String and a std::function. Queue::pop() copied the item out but left the slot
populated, so all RAPI_MAX_COMMANDS slots permanently retained a String and a
callback on the heap. Every time a slot was reused those two blocks were freed
and re-allocated wherever the allocator happened to be, walking steadily
further into the largest free run.

The result is fragmentation rather than a leak, so it is invisible to a
free-heap check. Measured on an ESP32 sending 1.3 RAPI commands/sec, the
largest contiguous block fell from ~57KB to ~2KB over two days while total free
heap held at 70-85KB, ending in a panic reboot. An otherwise identical unit
whose controller link was down held 55KB for 15 hours. Anything needing a large
buffer -- TLS, OTA -- fails long before the heap looks full.

Three changes:

- Queue::pop() releases the slot, so a popped item is not kept alive by the
  queue. push() takes a const reference rather than a copy.
- CommandItem::command becomes a fixed char[]. Commands longer than the buffer
  are rejected with the already-defined RAPI_RESPONSE_CMD_TOO_LONG instead of
  being truncated -- a truncated "$SC 32" is a valid command with a different
  meaning.
- The completion callback gets fixed inline storage rather than std::function,
  which heap-allocates any callable above its 8-byte small-object buffer. The
  library's callbacks are [this, callback] lambdas at 20 bytes, so every queued
  command allocated. A callable too large to fit inline is now a compile error
  rather than a silent return to the heap.

Costs RAPI_MAX_COMMANDS * ~140 bytes of .bss, which cannot fragment, in place
of the heap blocks that could.
onEvent() read tokens 1..4 without checking the token count. getToken()
returns NULL at or past the count, and strtol(NULL) dereferences it, so an
async frame carrying fewer fields than expected panics the host rather than
being ignored. The checksum is verified before this runs, so this is not
corruption -- it is a controller sending a shorter frame than assumed, which a
different firmware version legitimately can.

Every branch now checks the count first, as the request handlers elsewhere in
this file already do. $AB requires three tokens rather than two because the
firmware string is passed to the boot callback and consumers dereference it
without a NULL check.

Also report failures instead of dropping them: each public method began with
`if (!_sender) { return; }`, which abandons the caller's callback entirely.
A caller that sets a pending flag and waits for completion waits forever, with
no result and no error. They now complete with a new RAPI_RESPONSE_NOT_CONNECTED.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant