Skip to content

fix(aws-serverless): Keep the Lambda extension polling past 300s invocations - #24219

Open
LuccaRebelloToledo wants to merge 1 commit into
getsentry:developfrom
LuccaRebelloToledo:fix/lambda-extension-poll-timeout
Open

fix(aws-serverless): Keep the Lambda extension polling past 300s invocations#24219
LuccaRebelloToledo wants to merge 1 commit into
getsentry:developfrom
LuccaRebelloToledo:fix/lambda-extension-poll-timeout

Conversation

@LuccaRebelloToledo

@LuccaRebelloToledo LuccaRebelloToledo commented Sep 8, 2026

Copy link
Copy Markdown

The Lambda extension stops polling the Extensions API after any invocation that runs longer
than 300s, and every later invocation on that execution environment is then held open until the
function timeout kills it. Full mechanism, measurements and a repro that needs no AWS account
are in #24218.

Fixes #24218

Why http.request rather than lifting the limit on fetch

The 300s bound is undici's headersTimeout default. Raising it means passing a dispatcher,
which means depending on undici directly — node:http is already imported in this file, the
Extensions API is plain HTTP on localhost, and http.request has no default timeout.

Why the poll carries no deadline, and keep-alive instead

An earlier revision of this branch put a 960s deadline on the request, reasoning that it had to
sit above the 900s Lambda ceiling. That bound was wrong. The poll is open across the
environment's frozen idle time as well, which is unbounded — a warm environment can sit in
/event/next for hours. req.setTimeout runs on real time, unlike undici's tick-driven clock,
so a deadline would come due during the freeze and fire on thaw, destroying a poll that was
about to be answered and writing an error line on the first invocation after every long idle.

What the deadline was there for is a peer that goes away without a FIN/RST, and TCP keep-alive
covers that without bounding a legitimately long-open request — probes only travel while the
environment is running, so there is nothing to fire spuriously on thaw.

Why the retry is not unconditional, and why the fatal path exits

Ending the loop is what turns one failed request into an extension that is registered but
permanently silent, so a failed poll is retried. But retrying a non-transient 4xx never recovers
— it just buries the reason under a console error every few seconds for the life of the
environment — so those propagate. 408 and 429 are excluded, since those do start working again.
The check is structural on statusCode rather than instanceof, since a transport failure
carries code and the error may cross a module boundary.

What counts as a failed poll

run() decides when to stop from eventType, so anything it cannot read has to be a failure
rather than an event. A 200 whose body is not the event JSON used to become {}, which is
indistinguishable from an INVOKE — it reset the backoff, skipped the sleep and re-polled
immediately, so any endpoint answering 200 with non-JSON turned the loop into a tight silent
spin, and a SHUTDOWN delivered that way would never have stopped it.

Retries are also bounded. The backoff alone never terminates, so a runtime API that becomes
unreachable in a way that is not a 4xx would write one console error every 5s for as long as the
environment is thawed, for a condition that is not going to clear.

Why the fatal path exits

The fatal path has to exit. Logging and unwinding is not enough: the tunnel server holds
a referenced handle, so the process would stay alive and registered while never asking for
another event — which is precisely the hang this PR fixes. It reports error('exit', …) so
Lambda records it and recycles the environment, then exits non-zero.

The specific case that used to produce an unrecoverable synchronous throw on every iteration was
a registration that returned 200 without an identifier, leaving _extensionId null. That is now
rejected at registration instead.

Why SHUTDOWN handling is part of this

The extension registers for INVOKE and SHUTDOWN and ignores the event type. Polling a
runtime API that is being torn down only fails on the way out, which was harmless while a failed
poll ended the loop silently — but with retries added it would have written several console
errors on every execution environment teardown, for every user. next now returns the event so
the loop can stop.

That also removed the AbortSignal an earlier revision of this branch carried: reading the
lifecycle event is the right way to end the loop, and the signal was only ever test scaffolding.

Why console instead of the debug logger

debug is enabled from Sentry.init. The extension is a separate process that never calls it,
so DEBUG_BUILD && debug.error(...) in this file could not print under any option or env var —
which is a large part of why this went unnoticed. consoleSandbox matches what this file
already does for the DSN warning.

Verification

The A/B in #24218 was run three ways on public.ecr.aws/lambda/nodejs:22 with a 900s function
timeout and a reused execution environment: the published :88 layer and a yarn build:extension of develop both hang on the second invocation; this branch returns in ~110ms.

request and ExtensionsApiError are exported for the unit tests, matching the existing
getSentryDSNFromEnv precedent.

The exit is deferred by one turn of the event loop: process.exit does not wait for stderr,
which is a pipe under Lambda, so exiting straight out of the catch truncates the message it
just wrote to a single pipe buffer. Response bodies interpolated into error messages are
truncated for the same reason.

One test asserts that the poll does not go through fetch, which is an assertion about wiring
rather than behaviour. It is deliberate: the property it guards — that the poll has no 300s cap
— cannot be observed in CI, and without it nothing ties next to request. Happy to drop it if
you would rather not carry that shape.

@LuccaRebelloToledo
LuccaRebelloToledo force-pushed the fix/lambda-extension-poll-timeout branch from d38c2a3 to ba75461 Compare September 8, 2026 22:05
@LuccaRebelloToledo
LuccaRebelloToledo marked this pull request as ready for review September 8, 2026 22:05
@LuccaRebelloToledo
LuccaRebelloToledo requested a review from a team as a code owner September 8, 2026 22:05
@LuccaRebelloToledo
LuccaRebelloToledo requested review from JPeer264 and isaacs and removed request for a team September 8, 2026 22:05
@LuccaRebelloToledo
LuccaRebelloToledo marked this pull request as draft September 8, 2026 22:06
@LuccaRebelloToledo
LuccaRebelloToledo force-pushed the fix/lambda-extension-poll-timeout branch 2 times, most recently from 4a517e2 to 67b97e7 Compare September 8, 2026 22:36
…cations

`/event/next` acknowledges the previous event and waits for the next one, so the
poll stays open for the whole of the following invocation. Node's `fetch` caps
that at undici's 300s `headersTimeout`, and the rejection escapes a loop with no
`try`/`catch` — the extension never asks for another event, and Lambda holds
every later invocation on that execution environment until the function timeout.

The poll now uses `http.request`, which has no default timeout. It carries no
deadline either: the poll also spans the environment's frozen idle time, which is
unbounded, so a socket deadline would fire on thaw and destroy a poll that was
about to be answered. TCP keep-alive covers the case a deadline was there for.

A failed poll is retried with capped backoff, bounded so a failure that stops
recovering exits rather than logging every 5s forever. A refused poll — 4xx other
than 408 and 429 — and a body that is not the event JSON are both failures rather
than events, so neither resets the backoff or slips past the SHUTDOWN check.
Exiting reports to the Extensions API first, so Lambda recycles the environment
instead of leaving it registered and silent. Failures are reported through
`console`: `debug` is only enabled from `Sentry.init`, which this process never
calls.

Fixes getsentry#24218

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
@LuccaRebelloToledo
LuccaRebelloToledo force-pushed the fix/lambda-extension-poll-timeout branch from 67b97e7 to 3793baf Compare September 8, 2026 22:49
@LuccaRebelloToledo
LuccaRebelloToledo marked this pull request as ready for review September 8, 2026 22:59
@mydea
mydea requested a review from msonnb September 9, 2026 07:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Lambda extension stops polling after a 300s invocation, hanging every later invocation on that execution environment

1 participant