dyncfg: require a ParameterScope on every Config

[antiguru]

Motivation

The scope specifier introduced by the scoped feature flags work
(doc/developer/design/20260609_scoped_feature_flags.md) was an optional
.scoped(..) builder step on Config. A new dyncfg therefore defaulted
silently to environment-wide, and a config that is actually realized per replica
was easy to leave unannotated. That failure mode is invisible: the LaunchDarkly
sync loop never evaluates a replica context for an unannotated config, so a rule
targeting a replica or a size family is a no-op with no error anywhere.

Only 11 dyncfgs carried a scope. Several obviously replica-local ones did not,
including the whole lgalloc family apart from enable_lgalloc, the memory
limiter, and every storage-worker and persist-client config.

Description

Make scope the fourth argument of Config::new and drop Config::scoped, so
every declaration has to make the choice, then annotate all 276 existing dyncfgs
from their read sites: 184 replica-local, 92 environment-wide, none
cluster-coherent (no dyncfg feeds OptimizerFeatures, which is the only place a
cluster-scoped value is resolved today).

The rule, now written down on ParameterScope:

• A config realized inside a clusterd process is replica-local. That covers
  the compute worker config set, the storage worker config set, the persist
  client config set and mz_metrics — all four of which the per-replica dyncfg
  push reaches (handle_update_configuration applies the pushed updates to the
  last three; the storage worker's own set is served by the storage controller's
  layer). environmentd reading such a config for its own process is fine and
  sees the environment-wide value.
• A config environmentd resolves for one specific replica, whether it ships
  the value there or acts on it itself, is also replica-local — but only if
  the read site resolves that replica's overrides.
• A config realized in environmentd or balancerd with no single replica in
  scope is environment-wide.
• A config whose value must agree across the replicas of a cluster stays
  environment-wide even when it is read on clusterd.

environmentd read sites, where the overrides have to be constructed first

Seven configs are read in environmentd but resolved per replica. They now read
through that replica's overrides, which is what makes their Replica
declaration real rather than decorative:

• Controller::provision_replica builds a replica's TimelyConfig from
  arrangement_exert_proportionality, enable_timely_zero_copy,
  enable_timely_zero_copy_lgalloc and timely_zero_copy_limit.
• ComputeController::add_replica_to_instance freezes
  compute_replica_expiration_offset and
  enable_arrangement_dictionary_compression_alpha into ReplicaConfig.
• The controller's per-replica hydration interceptor enforces
  compute_hydration_concurrency. It takes a different route: rather than a
  point lookup, SequentialHydration keeps a config set of its own, fed from the
  CreateInstance and UpdateConfiguration commands it already absorbs. Those
  have been specialized for the replica by
  Instance::specialize_command_for_replica, so the set holds exactly what the
  replica itself reads. It has to be a set of its own: a cloned ConfigSet
  shares its values, so applying overrides to a clone would overwrite the
  environment-wide configuration for everyone.

Supporting changes: Config::get_with_overrides layers a replica's
ConfigUpdates over a ConfigSet (a mistyped override is logged and ignored
rather than panicking a controller path); the outer Controller keeps the pushed
override map and fans it out to the compute and storage controllers, so the
coordinator has a single push entry point; and the coordinator installs the map
before creating replicas during bootstrap, not only after, since the
provisioning-time values are frozen by then. The DDL path already pushed before
create_replica, so it needed no reordering.

Deliberate environment-wide declarations worth a reviewer's eye

• enable_compute_replica_expiration — the env-wide kill switch, read when
  specializing CreateInstance; compute_replica_expiration_offset is the
  replica-scoped half of the pair.
• compute_subscribe_snapshot_optimization — read at plan time and at render
  time, and it changes what a subscribe emits.
• storage_enforce_external_addresses — a security control; a per-replica
  override would weaken it for part of an environment.
• persist_use_critical_since_catalog, the catalog and expression-cache forced
  compaction knobs, txn_wal_apply_ensure_schema_match — read only from
  environmentd.

Verification

• cargo check --workspace --all-targets and cargo clippy --all-targets on the
  touched crates are clean, bin/fmt applied.
• New mz-dyncfg unit test for get_with_overrides: absent override, present
  override, and a type-mismatched override falling back to the set's value.
• New mz-compute-client unit test that the hydration interceptor's concurrency
  follows the replica's configuration, through both the CreateInstance
  snapshot and a later UpdateConfiguration, and that the environment-wide set
  is left untouched.
• Two mechanical audits over the annotated tree, both clean: no
  environment-scoped config is read from a clusterd-side crate except the two
  documented deliberate cases, and every replica-scoped config has a
  clusterd-side read or is one of the seven resolved in environmentd above.
• Behavior is unchanged while enable_scoped_system_parameters is off (the
  default): the sync loop evaluates no scoped contexts at all, so the wider
  Replica surface costs nothing until the feature is enabled.

Not covered by an automated test: the controller wiring for the six
provisioning-time and replica-creation-time configs. Exercising it needs a
ComputeController / orchestrator harness that does not exist today, so it is
verified by construction and by the get_with_overrides test underneath it.

🤖 Generated with Claude Code

https://claude.ai/code/session_01RZo8dwEyXbXwUq6wb7BkeU

[antiguru]

We should ensure that the optimizer overrides are scoped to their cluster.

[antiguru]

Why do we specialize here instead of within the replica task, or where it's created?

[antiguru]

Agreed, moved into the replica task in b23e77a.

ReplicaTask now owns a replica_dyncfg next to the environment-wide dyncfg, seeded at task creation and kept current by apply_config_command in the message loop, right where specialize_command already runs. SequentialHydration takes it as an argument to absorb_command / observe_response and holds no configuration of its own, so it cannot read the environment-wide value by accident. It reads the replica's or nothing.

Not at creation time, though: the task would then need the overrides plumbed into ReplicaClient::spawn as a second channel for something the command stream already carries, and it would go stale on the next push. Tracking the commands keeps one source of truth, and it is what the replica itself sees.

The test moved with the logic. replica_dyncfg_tracks_config_commands covers the seed, the CreateInstance snapshot, the UpdateConfiguration delta and the environment-wide set staying untouched. The interceptor's test is now behavioral rather than a getter check: at concurrency 1 the second Schedule is held back, and raising the concurrency in the supplied set releases it.

---

Generated by Claude Code

[antiguru]

Could this leak updates for replicas that are dropped?

[antiguru]

Yes. Fixed in b23e77a.

The assignment replaces the whole map, so it does not accumulate across pushes, and I had assumed that was enough. It is not: the coordinator only calls push_replica_dyncfg_overrides when replica_scoped_config_changed, so dropping a replica does not rebuild the map. A dropped replica's entry survives until the next scoped-config change, and an environment that churns replicas while the scoped config sits still accumulates them indefinitely.

Now pruned in the drop path instead of relying on the coordinator to re-push: Controller::drop_replica, ComputeController::drop_replica, compute Instance::remove_replica and storage Instance::drop_replica, which is all four places holding the map.

No correctness consequence that I can find, since replica IDs come from a monotonic sequence and are not reused, so a stale entry could never be picked up by a later replica. It was memory retention only.

---

Generated by Claude Code

[antiguru]

What happens if they don't agree?

[antiguru]

Nothing bad, which means my justification was wrong. Rewritten in b23e77a.

The two reads are asymmetric. In environmentd the flag gates whether optimize_dataflow_snapshot runs, which may clear with_snapshot on an import. On the replica, render.rs:279 computes if import.with_snapshot || !subscribe_snapshot_optimization { Include } else { Exclude }, so the replica-side read can only ever put a snapshot back, never take one away.

So both directions are safe. Flag on in environmentd and off on the replica means the snapshot is fetched anyway, correct but unoptimized. Flag off in environmentd and on at the replica means with_snapshot is still true, so the snapshot is included. Two replicas of one cluster disagreeing costs work, not correctness, since elision only happens where the optimizer proved the snapshot is not needed for the sink's output.

The real reason it is environment-scoped is the plan-time read: OptimizerConfig::from(&SystemVars) has no replica in scope, and no cluster either. Cluster would be the honest scope, but this is a plain OptimizerConfig field rather than an OptimizerFeature, and OptimizerFeatureOverrides is the only thing plan sites resolve cluster overrides through. persist_fast_path_order on the line above is in exactly the same position. Happy to file a follow-up for extending cluster resolution to the non-feature fields of OptimizerConfig if you think it is worth it.

---

Generated by Claude Code

[antiguru]

On the review summary, about the optimizer overrides being scoped to their cluster.

I checked every plan site, and the cluster-scoped optimizer overrides are resolved everywhere a cluster is in scope. cluster_scoped_optimizer_overrides(cluster_id) is applied in peek.rs (both paths), create_index.rs (all three), create_materialized_view.rs (all three), subscribe.rs (both), introspection.rs, coord.rs:3555 and frontend_peek.rs (both). The sites that do not resolve them are the ones with no cluster to resolve against: create_view.rs optimizes a view, which is not installed on a cluster.

Two gaps worth naming, neither introduced by this PR:

CatalogState MV rehydration (src/adapter/src/catalog/state.rs:1414) builds its OptimizerConfig with get_cluster(mv.cluster_id).config.features() but not the scoped layer, unlike every sequencer path above. A cluster-scoped override that changes an MV's plan would then have the rehydrated plan disagree with the one the sequencer produced. I have not traced how far that plan travels, so I am flagging rather than claiming a bug. Happy to dig in if useful.

Dyncfgs cannot be cluster-scoped at all today. That is why this PR annotates 276 dyncfgs as 184 replica and 92 environment, with zero cluster. Cluster resolution at plan time runs entirely through OptimizerFeatureOverrides, which covers OptimizerFeatures and nothing else, and no dyncfg feeds OptimizerFeatures. Two dyncfgs do reach the optimizer, as plain OptimizerConfig fields next to features: persist_fast_path_order and compute_subscribe_snapshot_optimization (optimize.rs:306-307). Both are conceptually cluster-coherent and both are forced to Environment, because declaring Cluster would be a silent no-op, exactly the failure mode this PR is about. Closing that gap means teaching the plan sites to resolve cluster overrides for the non-feature fields of OptimizerConfig too. Say the word and I will file it, or fold it into this PR if you would rather it not ship half-covered.

---

Generated by Claude Code