Skip to content

feat: declared background workers + frankenphp_get_worker_handle() - #2617

Open
nicolas-grekas wants to merge 37 commits into
php:mainfrom
nicolas-grekas:bgworker-server
Open

nicolas-grekas wants to merge 37 commits into
php:mainfrom
nicolas-grekas:bgworker-server

Conversation

@nicolas-grekas

@nicolas-grekas nicolas-grekas commented Aug 21, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #2664, without which a worker that fails to boot cannot be shut down: its commit comes first.

A background worker runs a PHP script in a loop outside the HTTP request cycle, on its own thread, sharing the runtime with the request threads: a queue consumer, a scheduler, a metrics pusher, anything that has to live as long as the server. It is declared like any other worker, with background; name is required, num defaults to one thread, and background threads come on top of num_threads and max_threads.

php_server {
	worker {
		file jobs.php
		name jobs
		background
	}
}

The script takes a handle on the worker and ticks it, waiting on it between calls through an Io\Poll\Context:

<?php

$handle = new FrankenPHP\WorkerHandle();

$poll = new Io\Poll\Context();
$poll->add($handle, [Io\Poll\Event::Read]);
// plus the handles the script waits on

while ($handle->tick()) {
    $poll->wait();

    doSomeWork();
}

Io\Poll comes with PHP 8.6, and the handle implements its Handle interface on every supported version: the poll API declares it there, FrankenPHP below it, so the loop above runs unchanged on 8.2 with symfony/polyfill-io-poll, which backs the context with stream_select().

tick() is where the runtime and the script meet, the background analog of frankenphp_handle_request(). Its first call marks the worker ready: the server start waits for it, max_execution_time bounds everything before it and nothing after, and an exit before it is a boot failure. It returns false once FrankenPHP drains the worker, on shutdown, reboot or restart, so the loop ends and the script returns. It never blocks and never hands out work.

getStream() hands out the stream the handle waits on, for the libraries that take one, Revolt and amphp among them, and for a script with no loop of its own, which can stream_select() on it or park in a blocking read. Waiting is all it supports: reading steals the bytes tick() would have consumed, and what it carries is not part of the contract. It is readable once at start, so a loop that waits on the handle becomes ready on its own.

The lifecycle mirrors HTTP workers: a cooperative exit re-runs the script, a crash restarts it with a capped quadratic backoff, and max_consecutive_failures fails Init() during startup only, since a background worker reaches its ready point on its own and would otherwise spin. Worker names are scoped like paths, unique within a php_server or among global workers, and metrics carry worker and server labels.

On top of this: shared state in #2635, tasks in #2636, their metrics in #2637.

@henderkes

Copy link
Copy Markdown
Contributor

Please rewrite the PR description to not be LLM slop reasoning with itself about what it did and why. I've tried reading this three times and I just can't.

@nicolas-grekas

nicolas-grekas commented Aug 22, 2026 •

Copy link
Copy Markdown
Contributor Author

Sure, I'll let you know when I'm done, for now I just let it do the rebase 😅

@henderkes henderkes left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens here when a global background worker and a php_server scoped background worker share the same name and are both eligible for the same source file?

Comment thread frankenphp.c Outdated
Comment thread threadbackgroundworker.go Outdated
Comment thread phpthread.go Outdated

@henderkes henderkes left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

found another one, anyway, have you tested this on windows?

Comment thread frankenphp.c Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds declared background PHP workers with graceful stop-stream handling and Caddy configuration support.

Changes:

  • Adds background-worker lifecycle, validation, and thread allocation.
  • Exposes frankenphp_get_worker_handle().
  • Adds Caddy integration, documentation, fixtures, and tests.

Reviewed changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
worker.go Registers and validates background workers.
threadbackgroundworker.go Implements background-worker lifecycle.
requestoptions.go Rejects background workers for HTTP requests.
phpthread.go Drains handlers during shutdown and transitions.
phpmainthread.go Drains handlers during reboot.
options.go Adds WithWorkerBackground().
frankenphp.go Reserves background-worker threads.
frankenphp.c Implements stop pipes and PHP API.
frankenphp.h Declares C primitives.
frankenphp.stub.php Declares the PHP function.
frankenphp_arginfo.h Registers generated arginfo.
docs/config.md Documents background configuration.
caddy/workerconfig.go Parses background worker blocks.
caddy/config_test.go Tests Caddy parsing and validation.
bgworker_test.go Tests lifecycle, restart, scope, and validation.
testdata/bgworker/basic.php Provides lifecycle fixture.
testdata/bgworker/crash.php Provides restart fixture.
testdata/bgworker/early-return.php Provides startup-failure fixture.
testdata/bgworker/named.php Provides named-worker fixture.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread frankenphp.c
Comment thread threadbackgroundworker.go Outdated
Comment thread threadbackgroundworker.go Outdated
Comment thread frankenphp.go Outdated
Comment thread testdata/bgworker/crash.php Outdated
@nicolas-grekas
nicolas-grekas force-pushed the bgworker-server branch 7 times, most recently from ac0896d to 3e93a24 Compare September 6, 2026 16:35
@nicolas-grekas

Copy link
Copy Markdown
Contributor Author

Two review-level items.

Name collision between a global and a php_server background worker: names are now scoped like paths, unique within a php_server or among global workers, so both start. Each script sees jobs in FRANKENPHP_WORKER, WithWorkerName() resolves within the request's server first, and metrics and logs report the scoped one as <server name>:jobs (server names get a numeric suffix when two blocks resolve to the same one). Two workers with the same name inside one php_server fail Init(). TestBackgroundWorkerOnServer and same_worker_name_in_two_servers cover both directions.

Windows: the Windows workflow runs the full suite on PRs and it passes here on 8.5.10, background worker tests included. It also surfaced that stream_select() on a pipe would have spun on 8.4 and older, since php_select() only waits properly on sockets before the GH-16889 fix, which is 8.5 only. The handle is therefore backed by a socket pair on every platform, which takes the version-independent Winsock path.

The branch is squashed to 3e93a24; sha references in earlier replies predate the squash.

@nicolas-grekas

Copy link
Copy Markdown
Contributor Author

Since the replies above, a self-review pass amended into the single commit (2f9c5b6):

  • a worker scoped to a Server never passed to WithServer() panicked on a nil map; Init() now rejects it
  • failureCount reset at the ready point, like HTTP workers: a crash after it restarts right away, only boot failures count toward max_consecutive_failures
  • the drain race check in setupScript also covers TransitionRequested
  • Windows: PHP's socketpair() emulation listens on INADDR_ANY and accepts the first peer, so the pair is verified with getpeername/getsockname; SOCK_CLOEXEC where available
  • a warning after 10s when a script has not waited on its handle, since Init() and Shutdown() wait for that point; docs say to block on the handle, feof() polling is not a wait
  • max_threads is rejected on background workers instead of ignored, and the Caddyfile requires num
  • untyped resource return in the stub, dead zend_unset_timeout() between requests removed, background threads no longer reported busy by the threads endpoint
  • tests: parking on a blocking read, RestartWorkers() draining a parked script, the handle throwing outside a background worker, the new validations

CI: all test jobs pass. The Windows job's caddy-suite timeout (POSTed configuration isn't active, then an admin GET hangs) is the same flake main's nightly runs hit (33416988237, 33302318173) and passes on rerun. The Docker matrix fails on the Go 1.27 runtime/cgo link error that #2634 and main's nightly show too.

nicolas-grekas added a commit to nicolas-grekas/frankenphp that referenced this pull request Sep 7, 2026
The task half of php#2319, on top of the background workers and their shared
vars: a request, an HTTP worker or another background worker hands work to
a named background worker with frankenphp_send_task(), which returns a
stream carrying the updates the worker sends back with
frankenphp_update_task(). The worker dequeues tasks with
frankenphp_receive_task() after reading a "task\n" line on its handle: the
one handle of php#2617 carries both the drain EOF and the wake-ups, so a
script keeps a single stream_select() loop. The line is a wake-up, not a
count: every thread of a pool gets one per task, the first one back in its
loop takes the task and the others get null.

send_task() blocks until a thread of the worker picks the task up and
throws on timeout, so a busy worker pushes back on its senders instead of
queueing without bounds; tasks queued while a thread restarts are signaled
again on its next run. Names resolve like frankenphp_get_vars() does.

Each task gets a socket pair. The sender's stream is a socket stream over
one end, one byte per update and EOF at completion, so stream_select()
bounds the wait or multiplexes tasks, and a blocking read parks as well;
closing it abandons the task. The receiver's stream is a socket stream
over the other end: updates go through update_task(), the stream itself
reports the sender's close as EOF to stream_select() and feof(), so a long
task learns that nobody waits for its result, and update_task() throws.
Closing it completes the task, unless the close is the resource cleanup of
request shutdown, which means the script ended with the task open: the
sender's next read throws instead of returning null. Sixteen updates are
buffered per task, past that update_task() waits for the sender to read.

Payloads and updates follow the set_vars() whitelist and travel as
persistent tables through the Go side, which owns them until they are
copied into request memory. The streams reference their task through a
cgo handle; the task is freed once both sides closed, or by the sender
when no thread picked it up. The stop sockets of a worker's threads are now
guarded by its task queue mutex, since senders write to them.

Compared to php#2319: no queue ahead of pickup and no cancellation before it,
no dedicated signaling stream, no global task table.
nicolas-grekas added a commit to nicolas-grekas/frankenphp that referenced this pull request Sep 7, 2026
The task half of php#2319, on top of the background workers and their shared
vars: a request, an HTTP worker or another background worker hands work to
a named background worker with frankenphp_send_task(), which returns a
stream carrying the updates the worker sends back with
frankenphp_update_task(). The worker dequeues tasks with
frankenphp_receive_task() after reading a "task\n" line on its handle: the
one handle of php#2617 carries both the drain EOF and the wake-ups, so a
script keeps a single stream_select() loop. The line is a wake-up, not a
count: every thread of a pool gets one per task, the first one back in its
loop takes the task and the others get null.

send_task() blocks until a thread of the worker picks the task up and
throws on timeout, so a busy worker pushes back on its senders instead of
queueing without bounds; tasks queued while a thread restarts are signaled
again on its next run. The wait also ends when the sender's own thread is
drained for a restart or the shutdown, since the target's threads are
drained too. Names resolve like frankenphp_get_vars() does.

Each task gets a socket pair. The sender's stream is a socket stream over
one end, one byte per update and EOF at completion, so stream_select()
bounds the wait or multiplexes tasks, and a blocking read parks as well;
closing it abandons the task. The receiver's stream is a socket stream
over the other end: updates go through update_task(), the stream itself
reports the sender's close as EOF to stream_select() and feof(), so a long
task learns that nobody waits for its result, and update_task() throws.
Closing it completes the task, unless the close is the resource cleanup of
request shutdown, which means the script ended with the task open: the
sender's next read throws instead of returning null. Sixteen updates are
buffered per task, past that update_task() waits for the sender to read.

Payloads and updates follow the set_vars() whitelist and travel as
persistent tables through the Go side, which owns them until they are
copied into request memory. The streams reference their task through a
cgo handle; the task is freed once both sides closed, or by the sender
when no thread picked it up. The stop sockets of a worker's threads are now
guarded by its task queue mutex, since senders write to them.

Compared to php#2319: no queue ahead of pickup and no cancellation before it,
no dedicated signaling stream, no global task table.
nicolas-grekas and others added 30 commits September 24, 2026 09:54
Two paths the suite took for granted. A worker parked on its handle must
survive default_socket_timeout as well as max_execution_time, so the
fixture that disables neither now runs with both set to one second. And a
run gets one handle: the second fetch is the same stream, a fetch after
closing it is a fresh one, and the drain still reaches the script through
that one.
The directive, the Go option and the docs section carry the flag, the
function did not.
…d workers

Readiness rode on intercepting every PHP path that waits on the handle:
the read op, the select cast, the transport receive missed at first, and
whatever a future PHP adds. The contract was also read three different
ways during review. It is now a call the script makes, the background
analog of frankenphp_handle_request(): the first frankenphp_worker_tick()
of a run marks the worker ready, every call returns false once the worker
is drained, and the read, cast and transport hooks are gone.

The tick never blocks and never hands out work. It consumes whatever the
runtime wrote on the handle to wake the script up, so the script only
ever selects on the handle, alone or with its own streams, and the
protocol on it stays private.
…ched

FRANKENPHP_WORKER stays what it always was, "1" in HTTP workers, and is
not set in background workers, where FRANKENPHP_WORKER_BACKGROUND holds
the declared name instead. A script serving both roles tests which of
the two is set. The removal of inherited values is gone with the change
that motivated it: nothing about HTTP workers moves in this PR anymore.
A script that registers its handle with an event loop and runs it only
ticks when the handle is readable, so it never became ready before the
drain and Init() waited for it. One wake-up written at run setup makes
such a loop tick on its own: readiness then means the loop serviced the
handle once. The first frankenphp_worker_tick() consumes it, and a
script parked in a blocking read without ticking now fails its boot
fast instead of hanging the start.
The shared lifecycle keeps its struct, embedded by both worker handlers;
the interface that named the one step they supply is gone, that step is
a parameter.
The channel was set to nil once Init() had decided, read without
synchronization by the handlers, and the send blocked on a buffer sized
to the thread count. A background worker can tick, exit and fail its
next boot while Init() finishes, which an HTTP worker cannot since it
blocks once ready: that exit could race the nil write, or block its
thread on a full buffer. The channel now stays, an atomic startup flag
gates the sends, and the send never blocks.
Until its first frankenphp_worker_tick(), a run is under the limit like
any request: a setup that outlives it ends as a boot failure, with the
backoff and the cap. The first tick disarms the timer, and nothing
re-arms it past that point, so the loop has no time limit, like the
CLI. This replaces the per-request ini override, which exempted the
bootstrap too.
Cancelling the handle's watcher only removes that one callback, and
run() keeps going while any other referenced watcher exists. A drained
worker has to leave its loop, which is the driver's stop().
… fire

The limit is PHP's: on Windows CI the busy bootstrap ran its full five
seconds without the timer ending it, so the test now runs only with the
Zend max execution timers of ZTS builds on Linux, where it passes.
A background worker serves no requests, so it does not scale with the
CPUs like an HTTP one and nearly every declaration wrote "num 1". It is
now the default, and declaring "background" is enough; a pool still asks
for the threads it wants.
The missing stop socket of a background worker is an invariant, not a
runtime error: the pair is opened before the script starts and closed at
the next run setup, so the two guards are asserts now. A thread reaching
the ready callback without a background handler would wait out Init()
silently, so it panics instead. The context of a background run is not a
dummy request, and the field says so. The scope of a name collision is a
local variable rather than a method, and the Caddyfile reference keeps
the short version of the "background" line, the long one lives in the
worker documentation.
A loop selecting on the handle blocks rather than spinning, because the
tick consumed the wake-ups, the one sent at start included. The fixture
polls the handle before and after a tick and the docs say so.
Two workers on one script, told apart by a matcher, are the documented
way to give slow endpoints their own thread pool. The Caddy module used
to make their generated names unique, this PR moved the collision check
into the core and dropped that, so the configuration stopped booting.

A name generated from the script path is not a declaration: it gets a
numeric suffix, as before. A declared name still collides, which is what
a background worker needs to keep its identity.
Packing the server into the worker name changed every label value of a
php_server worker, which breaks the dashboards and alerts built on them.
The two are separate labels now, worker="<name>" and server="<name>",
empty for a global worker, so a query on the worker name alone selects
that worker in every server and the values are the ones FrankenPHP
always reported.
- pace a run that ends right after its ready point, clean or crashed, and cut the wait short on drain
- route extension SendRequest() to its worker directly and make SendMessage() fail after Shutdown()
- keep the declared path as the default name of a global Caddy worker
- guard the background run context with contextMu
- one drain owner on phpThread, one background TLS reset in C, the public read-timeout stream option
- docs: metric labels, the platform condition of the bootstrap bound, stream_select() and FD_SETSIZE
A worker without a name is reported under the path of its script, which
newWorker() resolved through symlinks. Deployments that publish releases
behind a symlink then move every worker label at each deploy, since the
resolved path names the release directory.

The default is now the path as declared, made absolute, for the Caddy
module and the Go API alike, so naming them in the module is no longer
needed; it would also make two workers sharing a script collide, where
an undeclared name gets a numeric suffix instead.
Every exit past the ready point was counted toward the restart backoff
unless the run outlived its one second cap, so a worker processing a
batch and returning, which is how a script keeps its memory fresh, was
throttled to one run per second after four of them.

A run now counts only when it ended too fast to have done anything, a
tenth of a second, which is what tells a spinning script from a working
one. A script returning at once is still paced the same way.
Splitting the identity of a worker into two labels changed the signature
of every worker method of the Metrics interface, which an implementation
living outside this repository has to follow.

Those methods keep the single identifier they always took. One method
carries the labels instead: DeclareWorker() names them once, before
anything else mentions the worker, and the Prometheus implementation
resolves the identifier through them. An outside implementation adds
that method, empty when it has no use for the labels, and keeps the rest
untouched.

The identifier is the qualified name again, so two workers that would
report under one are rejected at startup, as before.
… interface

DeclareWorker() restored the signatures of the worker methods but still added a method to Metrics, which an implementation outside this repository has to grow before it compiles again.

Metrics is now the interface it was, a worker scoped to a server being reported as "<server name>:<name>" as before. An implementation that also satisfies ServerMetrics receives the two names apart, which is what PrometheusMetrics does to label its series; the runtime picks the right shape once, in WithMetrics().
frankenphp_get_worker_handle() and frankenphp_worker_tick() are gone,
replaced by FrankenPHP\WorkerHandle: tick() is the ready point and the
liveness check, getStream() the stream to wait on, isValid() whether the
run still holds its socket.

One object instead of two global functions, and a place for the task API
to land. Only waiting on the stream is supported, what it carries is not
part of the contract and tick() consumes it, so the descriptor stays out
of the contract. On PHP 8.6 the class can implement Io\Poll\Handle
without moving anything else, which is what the polling API discussion
asked for.

A run still has one stream whatever the number of handles, so a script
may take one wherever it needs it.
The cache that keeps a loop from growing the resource list of a run moves
from a thread-local slot to the handle that hands the stream out, where
the rest of a handle's state already lives. A handle gives the same stream
every time, a fresh one once the script closed it, and another handle has
its own over the same socket, which is harmless since the stream does not
own it and the drain reaches every one of them.

Nothing of it survives the run any more: the handle takes its stream with
it, so the thread no longer carries one to reset between runs. Asking a
throwaway handle for a stream in a loop stays flat, the object frees its
stream as it goes.
…ake one

The waiting a script does is shown with an Io\Poll\Context first, which is
what a background worker should reach for on 8.6 and, through the polyfill,
below it. The stream keeps its paragraph, as what an event loop takes and
as the fallback for a script with no loop of its own, with the FD_SETSIZE
ceiling of stream_select() named there rather than in the middle of the
explanation.
unserialize('O:23:"FrankenPHP\WorkerHandle":0:{}') builds one on a request
thread without calling the constructor, and tick() on it then panics
go_frankenphp_background_worker_ready() from a cgo callback, taking the
process down: the ZEND_ASSERT that stood there is compiled out of release
builds. getStream() handed out a stream over fd -1 the same way.

The class is now @not-serializable, which refuses that reconstruction, and
being internal and final with a create_object handler it was already out
of newInstanceWithoutConstructor()'s reach. The methods no longer take the
constructor's word for it either: both check the thread they run on and
throw, so the Go side keeps its invariant with nothing able to break it.
crashCount was reset by any run longer than 100ms, so a script exiting
non-zero after, say, 150ms restarted with no backoff at all, some seven
times a second, each one logging a warning and counting a crash. Only a
clean exit resets it now: a worker processing a batch and returning still
starts fresh, a crashing one is paced by the backoff however long it took
to fail.
SendMessage() refused a server that is not registered while SendRequest()
left it to Server.ServeHTTP(). Same ErrNotRunning either way, one less
thing to wonder about when reading the two next to each other.
PHP's socketpair() emulation is not one: it binds a listener to
INADDR_ANY, so the port is reachable from off the machine while the pair
forms, and hands back whichever connection arrives first. The pair is
built here instead, the way libevent and Tor do it: the listener takes
the loopback address alone and SO_EXCLUSIVEADDRUSE, and a connection is
kept only when its peer is the socket we connected with. Another process
racing a connect is dropped and the next one accepted, where the check we
had before failed the whole pair and left the worker to retry.
Init() waits for a background worker to reach its ready point, its first
WorkerHandle::tick(). On a build with Zend max execution timers,
max_execution_time ends a bootstrap that overstays; without them
FrankenPHP disables that limit, so a script that parks before ticking, or
one whose wake-up at start is lost, kept the server start waiting for
ever with a warning as the only trace.

boot_timeout bounds that wait, 30 seconds by default, the same figure
PHP's own max_execution_time uses for the bootstrap it does bound. The
worker is then drained and stopped through the boot-failure path it
already has, and Init() returns the name of the worker that never ticked.
Zero waits for ever, for whoever wants the old behaviour, and HTTP
workers are untouched.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants