fix: stop the RAPI command queue fragmenting the heap - #6
Open
RAR wants to merge 2 commits into
Open
Conversation
The command queue is a static array of CommandItem, each holding an Arduino String and a std::function. Queue::pop() copied the item out but left the slot populated, so all RAPI_MAX_COMMANDS slots permanently retained a String and a callback on the heap. Every time a slot was reused those two blocks were freed and re-allocated wherever the allocator happened to be, walking steadily further into the largest free run. The result is fragmentation rather than a leak, so it is invisible to a free-heap check. Measured on an ESP32 sending 1.3 RAPI commands/sec, the largest contiguous block fell from ~57KB to ~2KB over two days while total free heap held at 70-85KB, ending in a panic reboot. An otherwise identical unit whose controller link was down held 55KB for 15 hours. Anything needing a large buffer -- TLS, OTA -- fails long before the heap looks full. Three changes: - Queue::pop() releases the slot, so a popped item is not kept alive by the queue. push() takes a const reference rather than a copy. - CommandItem::command becomes a fixed char[]. Commands longer than the buffer are rejected with the already-defined RAPI_RESPONSE_CMD_TOO_LONG instead of being truncated -- a truncated "$SC 32" is a valid command with a different meaning. - The completion callback gets fixed inline storage rather than std::function, which heap-allocates any callable above its 8-byte small-object buffer. The library's callbacks are [this, callback] lambdas at 20 bytes, so every queued command allocated. A callable too large to fit inline is now a compile error rather than a silent return to the heap. Costs RAPI_MAX_COMMANDS * ~140 bytes of .bss, which cannot fragment, in place of the heap blocks that could.
onEvent() read tokens 1..4 without checking the token count. getToken()
returns NULL at or past the count, and strtol(NULL) dereferences it, so an
async frame carrying fewer fields than expected panics the host rather than
being ignored. The checksum is verified before this runs, so this is not
corruption -- it is a controller sending a shorter frame than assumed, which a
different firmware version legitimately can.
Every branch now checks the count first, as the request handlers elsewhere in
this file already do. $AB requires three tokens rather than two because the
firmware string is passed to the boot callback and consumers dereference it
without a NULL check.
Also report failures instead of dropping them: each public method began with
`if (!_sender) { return; }`, which abandons the caller's callback entirely.
A caller that sets a pending flag and waits for completion waits forever, with
no result and no error. They now complete with a new RAPI_RESPONSE_NOT_CONNECTED.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
On an ESP32 running the OpenEVSE WiFi firmware, the largest contiguous free
heap block falls steadily until large allocations start failing, ending in a
panic reboot. Total free heap barely moves the whole time, so a
free_heapcheck shows nothing wrong — this is fragmentation, not a leak:
free_heapAnything needing a large contiguous buffer — TLS, OTA — fails long before the
heap looks full. OTA in particular starts failing partway through the download.
Root cause
The command queue is a static array of
CommandItem, each holding an ArduinoStringand astd::function.Queue::pop()copies the item out but leavesthe slot populated:
So all
RAPI_MAX_COMMANDSslots permanently retain aStringand a callback onthe heap. Each time a slot is reused, those two blocks are freed and
re-allocated wherever the allocator happens to be — walking steadily further
into the largest free run. They act as a set of slowly migrating pins that
ratchet the largest contiguous block downward.
The callbacks land on the heap because
std::functionallocates for anycallable above its small-object buffer (8 bytes here), and this library's
callbacks are
[this, callback]lambdas at 20 bytes. Every queued commandallocated.
Confirmed by walking the heap block layout with
heap_caps_walk(): a fewkilobytes of 20-byte blocks, scattered through what had been a 58 KB free run,
cut it to 31 KB. Symbolizing them (a captured
thispointing into the globalOpenEVSEobject, plus the manager function pointers) identified them as theOpenEVSEClass::begin/EvseMonitor::begincompletion lambdas.A natural control: two otherwise identical units, same firmware. The one whose
controller link was down — no sustained RAPI traffic — held its largest block at
55,284 bytes for 15 hours. The one sending 1.3 commands/sec collapsed to 2 KB.
Changes
Queue::pop()releases the slot, so a popped item is not kept alive by thequeue.
push()takes aconstreference rather than a copy.CommandItem::commandbecomes a fixedchar[]. Commands longer than thebuffer are rejected with the already-defined
RAPI_RESPONSE_CMD_TOO_LONGrather than truncated — a truncated
$SC 32is a valid command with adifferent meaning.
std::function.A callable too large to fit is now a compile error, so the failure mode is
a build break rather than a silent return to heap allocation.
Trades
RAPI_MAX_COMMANDS * ~140bytes of.bss, which cannot fragment, forthe heap blocks that could. No public API change; all existing call sites
compile unmodified.
Second commit: async event guards
Found while auditing the same paths.
onEvent()read tokens 1..4 withoutchecking the token count.
getToken()returnsNULLat or past the count andstrtol(NULL)dereferences it, so an async frame with fewer fields thanexpected panics the host. The checksum is verified before this runs, so this is
not corruption — it is a controller sending a shorter frame than assumed, which
a different firmware version legitimately can. Every branch now checks the count
first, as the request handlers elsewhere in the file already do.
$ABrequiresthree tokens rather than two because the firmware string is handed to the boot
callback and consumers dereference it without a NULL check.
Also in that commit: every public method began with
if (!_sender) { return; },abandoning the caller's callback. A caller that sets a pending flag and waits
for completion waited forever, with no result and no error. Those now complete
with a new
RAPI_RESPONSE_NOT_CONNECTED.Validation
Built and run on an ESP32 (
openevse_wifi_tft_v1). RAPI functionally intact —1,581 commands with a single failure (the usual boot-time timeout), live voltage
and temperature reading back from the controller normally.
Largest free block, same unit, same workload:
Before, it only ever fell. After, freed blocks coalesce again and the region
that was being carved up stays clean.
Known hazards not addressed here
Flagging rather than fixing, since both are larger decisions than this change
should make:
_commandComplete()keeps_waitingForReplyset across the handler call, soa handler that itself calls
sendCmdSync()/flush()re-entersloop()andruns the same handler a second time — after which
_sendNextCmd()reassigns_completeHandlerwhile itsoperator()is still on the stack. No currentcaller does this. Fixing it properly means deciding whether a nested
synchronous send should be allowed at all.
getToken()returns interior pointers into a buffer the next responseoverwrites, so callbacks must copy anything they keep. Currently undocumented.
A follow-up branch collects several smaller items from the same audit
(
sendCmdSyncdiscarding its timeout argument,OPENEVSE_VFLAG_DEFAULTexpanding to undefined
ECVF_*names,getCurrentCapacity's declared parameternames not matching the order actually passed). Kept separate so this change
stays reviewable on its own.