Skip to content

fix(multigres): stop base config data_directory from overriding the pooler data dir - #2344

Merged
mkindahl merged 1 commit into
developfrom
fix/multigres-data-directory-override
Aug 6, 2026
Merged

fix(multigres): stop base config data_directory from overriding the pooler data dir#2344
mkindahl merged 1 commit into
developfrom
fix/multigres-data-directory-override

Conversation

@mkindahl

@mkindahl mkindahl commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Problem

The Multigres pooler never bootstraps. pgctld's transient PostgreSQL (started during InitDataDir) dies at startup and never becomes ready, so the first-backup bootstrap retries forever (observed 791× over ~8h in one pod) and the pooler socket never appears — surfacing downstream as MonitorPostgres: failed to create first backup and failed to connect to Unix socket /var/lib/pooler/pg_sockets/.s.PGSQL.5432.

The pgctld logs only show the swallowed transient PostgreSQL did not become ready after 30 seconds. The real reason is in $PGDATA/setup.log:

FATAL:  data directory "/var/lib/postgresql/data" has invalid permissions
DETAIL:  Permissions should be u=rwx (0700) or u=rwx,g=rx (0750).

Cause

pgctld runs initdb at <pooler-dir>/pg_data and pins the data directory via -D/PGDATA. But the operator appends include = '/etc/postgresql/postgresql.conf' onto pgctld's generated config (via POSTGRES_INITDB_EXTRA_CONF). In the layered image that base config has an active data_directory = '/var/lib/postgresql/data', and because the include is last it wins — redirecting PostgreSQL off the freshly-initdb'd cluster onto the base image's empty, wrong-permission /var/lib/postgresql/data, which fails startup.

The pre-layered (non-layered) image shipped data_directory commented out, which is why it worked; the layered rewrite left it active. A comment in Dockerfile-multigres even asserted the base config "is never loaded" — that assumption is wrong (it is loaded, via the operator's include), which is the root of the regression.

Fix

Comment out data_directory in the layered image's copy of /etc/postgresql/postgresql.conf (via the same sed step that already removes the dangling wal-g include), so the base-config include becomes harmless regardless of whether it arrives via flag or env var. The stale "never loaded" comment is corrected.

postgresql.conf.j2 is intentionally left untouched: it is shared by the base supabase image and the VM/ansible deployments, which run postgres -D /etc/postgresql and genuinely rely on data_directory to locate the data — commenting it there would break them. The fix belongs in the Multigres layer, where pgctld pins the data dir itself.

Validation

  • Reproduced the identical FATAL locally with a layered image + the base-config include.
  • Confirmed live in the broken staging pod: postgres -C data_directory resolves to /var/lib/postgresql/data despite PGDATA being elsewhere; an empty extra-conf (no include) inits cleanly.
  • With this fix applied, init succeeds with the include still present and data_directory resolves to the correct pooler data dir.

🤖 Generated with Claude Code

…r data dir

The Multigres pooler failed to bootstrap: the transient PostgreSQL that
pgctld starts during InitDataDir died at startup with

  FATAL: data directory "/var/lib/postgresql/data" has invalid permissions

so it never became ready, the first-backup bootstrap retried forever, and
the pooler socket never appeared.

pgctld runs initdb at <pooler-dir>/pg_data and pins the data dir via -D, but
the operator appends `include = '/etc/postgresql/postgresql.conf'` onto the
generated config (POSTGRES_INITDB_EXTRA_CONF). In the layered image that base
config carries an active data_directory = '/var/lib/postgresql/data', and
since the include is last it wins, redirecting PostgreSQL off the freshly
initdb'd data dir onto the base image's empty, wrong-permission directory.

The pre-layered image shipped data_directory commented, which is why it
worked. Comment it out in the layered image too (alongside the existing wal-g
include removal), and correct the now-wrong comment claiming the base config
is never loaded. postgresql.conf.j2 is left untouched, so the base supabase
image and the VM/ansible deploys (which run `postgres -D /etc/postgresql` and
rely on data_directory) are unaffected.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@mkindahl
mkindahl requested review from a team as code owners August 5, 2026 15:37
@mkindahl
mkindahl marked this pull request as draft August 5, 2026 15:39
@mkindahl
mkindahl marked this pull request as ready for review August 5, 2026 16:58
@mkindahl
mkindahl added this pull request to the merge queue Aug 6, 2026
Merged via the queue into develop with commit 60816b7 Aug 6, 2026
117 checks passed
@mkindahl
mkindahl deleted the fix/multigres-data-directory-override branch August 6, 2026 08:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants