fix(aws-serverless): Keep the Lambda extension polling past 300s invocations - #24219
Open
LuccaRebelloToledo wants to merge 1 commit into
Open
Conversation
LuccaRebelloToledo
force-pushed
the
fix/lambda-extension-poll-timeout
branch
from
September 8, 2026 22:05
d38c2a3 to
ba75461
Compare
LuccaRebelloToledo
marked this pull request as ready for review
September 8, 2026 22:05
LuccaRebelloToledo
requested review from
JPeer264 and
isaacs
and removed request for
a team
September 8, 2026 22:05
LuccaRebelloToledo
marked this pull request as draft
September 8, 2026 22:06
LuccaRebelloToledo
force-pushed
the
fix/lambda-extension-poll-timeout
branch
2 times, most recently
from
September 8, 2026 22:36
4a517e2 to
67b97e7
Compare
…cations `/event/next` acknowledges the previous event and waits for the next one, so the poll stays open for the whole of the following invocation. Node's `fetch` caps that at undici's 300s `headersTimeout`, and the rejection escapes a loop with no `try`/`catch` — the extension never asks for another event, and Lambda holds every later invocation on that execution environment until the function timeout. The poll now uses `http.request`, which has no default timeout. It carries no deadline either: the poll also spans the environment's frozen idle time, which is unbounded, so a socket deadline would fire on thaw and destroy a poll that was about to be answered. TCP keep-alive covers the case a deadline was there for. A failed poll is retried with capped backoff, bounded so a failure that stops recovering exits rather than logging every 5s forever. A refused poll — 4xx other than 408 and 429 — and a body that is not the event JSON are both failures rather than events, so neither resets the backoff or slips past the SHUTDOWN check. Exiting reports to the Extensions API first, so Lambda recycles the environment instead of leaving it registered and silent. Failures are reported through `console`: `debug` is only enabled from `Sentry.init`, which this process never calls. Fixes getsentry#24218 Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
LuccaRebelloToledo
force-pushed
the
fix/lambda-extension-poll-timeout
branch
from
September 8, 2026 22:49
67b97e7 to
3793baf
Compare
LuccaRebelloToledo
marked this pull request as ready for review
September 8, 2026 22:59
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Lambda extension stops polling the Extensions API after any invocation that runs longer
than 300s, and every later invocation on that execution environment is then held open until the
function timeout kills it. Full mechanism, measurements and a repro that needs no AWS account
are in #24218.
Fixes #24218
Why
http.requestrather than lifting the limit onfetchThe 300s bound is undici's
headersTimeoutdefault. Raising it means passing a dispatcher,which means depending on
undicidirectly —node:httpis already imported in this file, theExtensions API is plain HTTP on localhost, and
http.requesthas no default timeout.Why the poll carries no deadline, and keep-alive instead
An earlier revision of this branch put a 960s deadline on the request, reasoning that it had to
sit above the 900s Lambda ceiling. That bound was wrong. The poll is open across the
environment's frozen idle time as well, which is unbounded — a warm environment can sit in
/event/nextfor hours.req.setTimeoutruns on real time, unlike undici's tick-driven clock,so a deadline would come due during the freeze and fire on thaw, destroying a poll that was
about to be answered and writing an error line on the first invocation after every long idle.
What the deadline was there for is a peer that goes away without a FIN/RST, and TCP keep-alive
covers that without bounding a legitimately long-open request — probes only travel while the
environment is running, so there is nothing to fire spuriously on thaw.
Why the retry is not unconditional, and why the fatal path exits
Ending the loop is what turns one failed request into an extension that is registered but
permanently silent, so a failed poll is retried. But retrying a non-transient 4xx never recovers
— it just buries the reason under a console error every few seconds for the life of the
environment — so those propagate. 408 and 429 are excluded, since those do start working again.
The check is structural on
statusCoderather thaninstanceof, since a transport failurecarries
codeand the error may cross a module boundary.What counts as a failed poll
run()decides when to stop fromeventType, so anything it cannot read has to be a failurerather than an event. A 200 whose body is not the event JSON used to become
{}, which isindistinguishable from an INVOKE — it reset the backoff, skipped the sleep and re-polled
immediately, so any endpoint answering 200 with non-JSON turned the loop into a tight silent
spin, and a SHUTDOWN delivered that way would never have stopped it.
Retries are also bounded. The backoff alone never terminates, so a runtime API that becomes
unreachable in a way that is not a 4xx would write one console error every 5s for as long as the
environment is thawed, for a condition that is not going to clear.
Why the fatal path exits
The fatal path has to exit. Logging and unwinding is not enough: the tunnel server holds
a referenced handle, so the process would stay alive and registered while never asking for
another event — which is precisely the hang this PR fixes. It reports
error('exit', …)soLambda records it and recycles the environment, then exits non-zero.
The specific case that used to produce an unrecoverable synchronous throw on every iteration was
a registration that returned 200 without an identifier, leaving
_extensionIdnull. That is nowrejected at registration instead.
Why SHUTDOWN handling is part of this
The extension registers for
INVOKEandSHUTDOWNand ignores the event type. Polling aruntime API that is being torn down only fails on the way out, which was harmless while a failed
poll ended the loop silently — but with retries added it would have written several console
errors on every execution environment teardown, for every user.
nextnow returns the event sothe loop can stop.
That also removed the
AbortSignalan earlier revision of this branch carried: reading thelifecycle event is the right way to end the loop, and the signal was only ever test scaffolding.
Why
consoleinstead of the debug loggerdebugis enabled fromSentry.init. The extension is a separate process that never calls it,so
DEBUG_BUILD && debug.error(...)in this file could not print under any option or env var —which is a large part of why this went unnoticed.
consoleSandboxmatches what this filealready does for the DSN warning.
Verification
The A/B in #24218 was run three ways on
public.ecr.aws/lambda/nodejs:22with a 900s functiontimeout and a reused execution environment: the published
:88layer and ayarn build:extensionofdevelopboth hang on the second invocation; this branch returns in ~110ms.requestandExtensionsApiErrorare exported for the unit tests, matching the existinggetSentryDSNFromEnvprecedent.The exit is deferred by one turn of the event loop:
process.exitdoes not wait for stderr,which is a pipe under Lambda, so exiting straight out of the
catchtruncates the message itjust wrote to a single pipe buffer. Response bodies interpolated into error messages are
truncated for the same reason.
One test asserts that the poll does not go through
fetch, which is an assertion about wiringrather than behaviour. It is deliberate: the property it guards — that the poll has no 300s cap
— cannot be observed in CI, and without it nothing ties
nexttorequest. Happy to drop it ifyou would rather not carry that shape.