Update trigger - #4
Open
tylerc-govsignals wants to merge 1794 commits into
Open
tylerc-govsignals wants to merge 1794 commits into
tylerc-govsignals wants to merge 1794 commits into
Conversation
…ids (#4770) Skew protection resolves a run's worker by (environmentId, externalId, status=DEPLOYED). A miss parks the run and then expires it, so deployments predating the feature — which already carry the same value in commitSHA — need externalId populated to stay reachable. Vercel instant-rollback is the sharpest case, which is why the scope is the current promotion plus a recent window rather than current alone. Follows the existing backfill shape: admin PAT, keyset cursor over environments, per-environment action results, pMap, dryRun defaulting to true. Reuses normalizeExternalDeploymentId so a backfilled id is byte-identical to what a build writes, and the update re-checks externalId IS NULL so a deploy landing mid-backfill keeps its own id. Refs TRI-13464.
…d shared test utilities (#4772) ## Summary Adds the read comparator for the in-progress migration of the run execution-snapshot log from Postgres to Redis. The comparator samples a single read against both stores, normalizes the two results to one shape, and reports any per-field difference with a tagged metric. It never serves a read itself: the diff layer imports only types, so it cannot hold a store client, and a test enforces that by failing if any value import appears. Also adds a combined Postgres-and-Redis test fixture and two shared test utilities (a cluster-slot assertion and a generic fault-injection harness) that the parallel Redis-store work reuses. Everything here is inert. Nothing constructs the comparator, so merging changes no runtime behavior. It becomes active only when a later change turns on compare mode. ## Notes The divergence classes separate real differences (scalar, ordering, waitpoint id set, validity, missing on one side) from two expected classes that must not be driven to zero: a rotated idempotency key, and a Redis-only surplus at a since-cursor tie. The since comparison is direction sensitive: a Postgres-only entry at the cursor is always a lost write, never an expected tie.
Adds an `ADMIN_DASHBOARD_ENABLED` env var (default: enabled) that turns the admin dashboard and user impersonation off for an entire instance. When disabled: - every admin dashboard page redirects away, and the admin navigation isn't rendered - existing impersonation cookies are ignored, and any lingering session is actively terminated with an audit record - every flow that could start an impersonation responds 404, and no impersonation tokens are minted Stopping an impersonation always works regardless of the flag, so nothing gets stuck. Machine-to-machine admin API endpoints are not affected. The variable is documented for self-hosters; instances that don't set it are unaffected.
…when a runs list query is too expensive (#4773) ## Summary When a runs list query is too expensive to complete, it now fails with a clear, actionable error instead of a generic 500. Previously, a runs list query that exceeded ClickHouse resource limits threw an opaque error. On the public `runs.list` API that surfaced as a retryable 500, so a customer task calling it would keep retrying a query that could never succeed. On the dashboard it rendered as a generic error page with no hint about what to do. ## Fix The ClickHouse client now tags resource-limit failures (memory, time, rows, bytes) with their error type, and the runs repository maps those to a dedicated `RunsListQueryError` (HTTP 422). - `runs.list` API returns 422 with a message telling the user to narrow their `created_at` range, plus an `x-should-retry: false` header so the SDK does not retry it. - The dashboard runs list (and the errors, scheduled, standard-task, agents, and webhooks list views) render a shared error state with the same guidance, so a too-broad time filter is recoverable by the user.
Switching between deployments in the dashboard re-fetched the whole build log stream from record zero and re-rendered the list line by line every time. Logs are now cached per deployment for the lifetime of the tab: revisiting a deployment shows its logs immediately, and the stream is resumed from the next unread record rather than restarted. Finished deployments whose stream has been read through the `finalized` event are served entirely from the cache. ### Changes The stream/cache logic moved out of the route into a `useDeploymentLogs` hook. On each deployment switch it seeds state from the cache, resumes the S2 read session at `nextSeqNum`, and writes back on cleanup or natural session end. Completion is derived from the stream's own `finalized` event (plus a terminal deployment status), not from the session closing, so a session cut short by token expiry or a proxy cannot pin a truncated log in the cache. Memory is bounded by a small LRU (`deploymentLogsCache`): at most 20 deployments and 20,000 log lines in total, least recently viewed evicted first. The most recently viewed deployment is always kept, so a single very large log can temporarily exceed the line budget on its own. Records are batched into one state update per tick instead of one per line.
## What
Makes `RoutingRunStore` correct when the run-ops layer routes across
more than two Postgres stores. Today it routes between a gen-1 `new`
dedicated database and a `legacy` control-plane database; this
generalizes every routing policy to N shards while keeping the two-store
behaviour byte-identical.
The change sets the four routing decisions that were implicit in code
order, and fixes one hazard that failed silently:
- **Id → shard key.** The router resolves a shard key with
`resolveShard` instead of the binary residency classifier, so a gen-2 id
reaches its own shard through the keyed map.
- **Membership vs routing.** `#distinctStores` (one entry per physical
database, aliases excluded by a declared `aliasOf`) drives every sum,
probe, and merge; `#shards` drives routing. An aliased shard can no
longer make a sum count one database twice.
- **Probe order.** A keyless lookup stays a sequential short-circuit at
two stores; above two it fans out in parallel, picks by precedence,
tolerates a single down leg, and keeps the canonical not-found throw on
the legacy leg.
- **Precedence and duplicates.** One merge helper across all four merge
sites. A duplicate id confined to `{new, legacy}` stays silent (the
known drain-mirror case); any other cross-shard duplicate increments
`runops_shard_duplicate_id_total` and logs at error level.
- **Disjoint sum (the silent hazard).** `countPendingWaitpoints` and the
waitpoint collector now partition absent ids by shard and **union by
id** rather than summing counts. A drain-mirrored waitpoint on both
gen-1 stores is counted once, so a blocked run can no longer hang
forever on a double-counted pending waitpoint.
- **Waitpoint completion.** A gen-2 waitpoint completes on its own
shard, overriding the legacy pins; a cuid waitpoint keeps its two-member
gen-1-pair probe unchanged.
- **Fail-loud creates.** A create with no shard key throws instead of
silently defaulting to `new`. An id resolving to an unconfigured shard
throws instead of being dropped.
Two new counters are exported: `runops_shard_duplicate_id_total` and
`runops_waitpoint_probe_fallback_total`.
## Why it is safe to merge
With only `{new, legacy}` configured every generalized rule reduces to
today's behaviour. `resolveShard` returns exactly what the old
classifier returned for every id shape that exists today, and no gen-2
id is minted yet. The only intentional behaviour change is the fail-loud
create throw; an enumeration of production call sites confirmed no
caller trips it.
## Testing
- New container-free algebra suite (50 cases) over probe order,
precedence, the duplicate alarm, the disjoint-sum partition, the
waitpoint probes, and the fail-loud paths.
- New `runOpsStore.nShardMatrix.test.ts` runs a four-store matrix
(legacy + new + two gen-2 shards) against real Postgres containers: the
disjoint-sum union, the alias topology, cross-tree completion,
pagination merges, and mixed-id hydration.
- New `makeNShardRunOpsPostgresTest(k)` fixture in
`@internal/testcontainers`.
- Full run-store corpus green: 71 files, 480 tests. Typecheck, lint,
format, and knip all clean.
## Notes
- Draft: opened for review; not marking ready yet.
- No changeset or `.server-changes` file: internal routing
infrastructure, no user-visible behaviour change.
- TRI-13427.
Auto-scroll now only follows while you are at the bottom. Scrolling up pauses it; scrolling back to the bottom, or clicking the new scroll-to-bottom button in the log header, resumes it. When you are at the bottom the same button scrolls to the top. Switching to another deployment starts at the bottom again.
…4777) The environment variable key and value inputs did not set an autocomplete attribute, so browsers could offer to autofill or save typed values as saved credentials. This sets `autoComplete="off"` on those inputs in both the create and edit forms, matching the `autoComplete="off"` convention already used on the other credential-name inputs. `autoComplete="off"` is a best-effort hint. Browsers may still ignore it for password-typed fields, so this is defense-in-depth hardening, not a hard guarantee that a password manager cannot store the value.
…4764) Part of the RunOps N-way sharding work. This lets the webapp hold N run-ops stores, configured by a single `RUN_OPS_SHARDS` JSON descriptor, and routes to them through the existing keyed router. **Inert with `RUN_OPS_SHARDS` unset** — the topology, the wiring and `ROUTING_ENABLED` are byte-identical to today. ## What's here - **`RUN_OPS_SHARDS`** — a zod-validated JSON array of shard descriptors (`key`, `region`, `url`, `replicaUrl`, `directUrl`, `replication`, `knobs`, `aliasOf`), validated at boot in the `parseMachinePresetCsv` style. Unset or `[]` → no shards. - **One run-ops client factory** — `buildRunOpsWriterClient`/`buildRunOpsReplicaClient` collapse into one `buildRunOpsClient` parameterized by role and resolved pool knobs. The control-plane builders (`buildWriterClient`/`buildReplicaClient`) are a separate path and stay untouched; every resolved value matches the former builders. - **Shard loop in `selectRunOpsTopology`** — one client pair per descriptor; an `aliasOf: "new"` descriptor reuses the new store's clients by reference and opens no pool. - **N-way `buildRunStore`** — builds N dedicated stores + the keyed router via a new `RoutingRunStore.fromShards`, keeping the two-store compat router when no shards are configured. - **`UnknownShardKey`** — raised when an id resolves to an unconfigured key; never falls back to another store. `fromShards` injects `resolveShard` so a gen-2 id routes to its own shard. - **Per-shard transaction resilience** — each shard gets its own retry budget. - **Mint bound** — `computeMintShard` intersects the active mint list with the configured descriptor keys, so a key with no descriptor is never minted into. - **Boot table** — logs `key`, address fingerprint (host:port/db, no credentials), and role, only when shards are configured. ## Ordering constraint Do **not** configure a `RUN_OPS_SHARDS` descriptor in any environment until the routing-semantics change (TRI-13427) lands — three fan-out sites still truncate at N>2. Merging this PR alone is safe (inert with the var unset); configuring a descriptor is what must wait. ## Testing - Run-store corpus: green with zero test-file diffs (the bit-identical proof for the compat router). - `runOpsDbTopology.test.ts` 17/17, `runStore.server.test.ts` 4/4, `runOpsMigration` family 149/149. - New unit suites: descriptor validation, pool-knob value tables, `fromShards` routing + `UnknownShardKey`, boot-table formatter, mint bound. - typecheck (webapp + run-store), knip, lint, format: pass. ## Changelog Internal run-ops sharding infrastructure. No changeset or `.server-changes`: the change is inert with `RUN_OPS_SHARDS` unset and has no user-visible behaviour. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Deployments currently leave little analytical trace. This PR makes every
deployment emit two analytics events to enable useful queries. It also
enables comparing deployments across build paths, CLI versions,
runtimes, and orgs.
### Where the events come from
```
trigger deploy
│
▼
initialize ─────────────────────────────▶ ✨ deployment.initialized
│ createdAt
▼
PENDING waiting for a build slot ┐
│ startedAt │ queue time
▼ ┘
INSTALLING build server installs deps ┐
│ installedAt (native paths only) │ install time
▼ ┘
BUILDING the image is built ┐
│ builtAt │ building time
▼ ┘
DEPLOYING indexing + registry push ┐
│ deployedAt / failedAt / canceledAt │ deploying time
▼ ┘
DEPLOYED · FAILED · TIMED_OUT · CANCELED
│
└───────────────────────────────────▶ ✨ deployment.finished
```
`deployment.finished` fires exactly once, whichever way the deployment
ends, and is backdated to cover the deployment's real lifetime. Not
every path visits every state (Depot deploys skip PENDING/INSTALLING,
for example) — a phase duration is simply omitted when its state was
never entered.
### What each event carries
- **Which path built it**: `depot`, `native`, or `native_local_bundle`
- **How it ended**: status, plus an error class and message when it
failed
- **How long each phase took**: queue, install, building, deploying, and
total — derived from the timestamps above
- **Who and with what**: org, project, environment, runtime, CLI
version, and how the deploy was triggered (CLI, GitHub, Vercel)
With that, one query gives failure rate per build path, duration
percentiles per phase, adoption per CLI version, or a per-org health
table.
### Fixes that ride along
- The old `deployment.outcome` span was silently dropped ~95% of the
time (it was subject to trace sampling). The new events opt out of
sampling explicitly, so every deployment is counted.
- The fail/timeout/finalize transitions were racy: a late timeout could
overwrite a successful deployment. They now use guarded writes, so
exactly one caller wins the terminal transition — and exactly one event
is emitted.
- Canceled deployments previously recorded nothing; they do now.
- The deployment's CLI version is now stored at initialization (new
nullable column), so even deploys that fail early are attributable to a
CLI release.
- Telemetry is flushed on shutdown (the last batch used to be lost on
every webapp deploy), and an optional second exporter can mirror just
these events into a dedicated dataset.
…bases (#4780) ## Summary The run-ops boot interlocks and the migration entrypoint each assume exactly two run-ops databases. This generalizes them to any number, so a deployment that configures `RUN_OPS_SHARDS` gets the same safety guarantees it gets today with two stores: no two stores may point at one database, every store that owns its own database must replicate to ClickHouse, and every store must have its schema migrated. With `RUN_OPS_SHARDS` unset, nothing changes. The distinctness check over a two-element set is the pairwise compare it replaces, replication coverage is the check it was, and the entrypoint runs the same two migration invocations. A shard may declare `aliasOf: "new"`, which shares an existing store's client by reference. An aliased shard is not its own database, so it is exempt from the distinctness check and needs no replication slot of its own. Every check keys that exemption on the declared field, never on client object identity: two client objects can sit over one database, which identity comparison cannot see. ## Design **Distinctness.** `probeDistinctDatabases` compared two URLs. It now delegates to `probeDistinctStores`, which reads every fingerprint in parallel and groups them by system identifier and database name. Any two stores under one key refuse the boot. The old pairwise entry point stays, so its existing container tests are the proof that set uniqueness over one pair gives the verdict it gave before. Fail-closed is unchanged: a probe that cannot answer returns not-distinct, because "distinct" is a positive claim a failed probe cannot support. **Co-residency.** The advisory runs once per store against the control plane. The legacy emission keeps its exact call shape and its untagged metric series, so an existing dashboard does not change. Each shard emits its own point carrying its shard key. Every store emits before any enforcement throw, so one offending store never costs another store its metric. **Replication.** `buildReplicationSources` appends one source per shard that owns its own database, taking the slot, publication and origin generation its descriptor declares. `assertReplicationCoversSplit` then requires a source per such shard. That check also closes a hole it inherited. The descriptor parser validates uniqueness among shards only, so a shard could take the slot name, publication name or origin generation of the legacy or the new source. The replication service does validate this, but it throws from its constructor, and the caller reaches that constructor only after shutting the bootstrap instance down: ```ts if (sources.length > 1) { await service.shutdown(); // legacy stream stops here service = new RunsReplicationService({ ... }); // throws: duplicate slotName } ``` The throw was not a `SplitReplicationMisconfiguredError`, so the process stayed up with no replication at all, legacy included, behind one logged line. That is the silent ClickHouse under-count the error exists to prevent. The check now runs at the boot gate, before anything is torn down, and raises a subclass the existing exit path already recognizes. A correct deployment already satisfies it, because two consumers on one WAL slot is a data race that cannot work. **Migrations.** Every shard runs the identical schema, so a new shard is the existing migrations against a new DSN. The runner image has no `jq`, so a small node script prints one DSN per line and the entrypoint loops over them. The loop is a `for` and not a `while read` pipeline: a pipeline subshell swallows a failed migration on any iteration but the last, which would let a broken shard boot. Tracing stays off across the capture and the loop, because `set -x` prints an assignment and a DSN carries credentials. Verified end to end against real Postgres containers for the fingerprint probes, and against the real shell block with a stubbed migration command: an aliased shard is skipped, `directUrl` wins over `url`, a failing shard stops the container on the first failure, and a malformed descriptor stops it before it migrates anything. Stacked on #4764. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…off-by-default dial (#4765) ## Summary Adds a `RunStore` decorator that mirrors execution snapshots into Redis alongside Postgres, plus the orphan-key sweep and the fault-injection suite that prove the write protocol converges after a crash. Nothing constructs it, so merging this changes no behaviour: the configuration, the production wiring and the Redis client all arrive in later work. The execution-state log is the hottest table in the run graph, and moving it out of Postgres has to happen without a big-bang cutover. This is the attachment point for that: a decorator that wraps the existing storage interface and intercepts only the methods that touch snapshots, so none of the many callers change. ## Design Write order is the correctness property, and the two orders differ on purpose. A transition writes Postgres first and Redis second. A crash in the gap leaves a run whose latest snapshot is stale, which is the state the heartbeat stall watchdog already heals in production today. A birth writes Redis first and Postgres second. A crash there leaves an unreachable key for a run that does not exist. Postgres first would instead leave a run with no snapshot at all, which the engine treats as a hard error, so the run would be stuck. Each order is chosen so the state a crash leaves behind is the harmless one. A lost cross-store write is never recovered by a transaction or an outbox; recovery is always the existing stall and repair job. A failed append retries, then hands the run to that job, and never rethrows, because Postgres has already committed and a throw would turn a healable gap into a caller-visible error. Inside a transaction the Redis half is staged and flushed only after the commit, so a rollback cannot leave Redis holding a transition that never happened. Reads are shape matched. Two of the snapshot reads take arbitrary Prisma arguments, and a key-value store cannot answer an arbitrary query, so the decorator recognises exactly the shapes the engine sends and delegates everything else. A miss falls back to Postgres, which is also how runs created before any cutover keep working. The sweep reaps under two rules, because neither can see what the other leaves behind. A finished run whose keyspace never received its completion expiry gets one applied. A keyspace with no run row at all, past an age threshold, is deleted; that is a crashed birth, which is non-terminal so it carries no expiry and has no run row, so the first rule can never match it. ## Inertness Three independent reasons this is a no-op if merged alone: - Nothing constructs the decorator or the Redis store outside tests. - No configuration reaches it, so the dial stays at its off position, which is a pass-through that makes no Redis call. - The existing Postgres store gains an off-by-default flag and two optional input fields. Both default to today's behaviour, and only the decorator would ever supply them. ## Notes for review The snapshot id and the creation instant are both minted by the decorator and written into both stores, so one snapshot has one identity and one timestamp wherever it is read. Without that, the two stores disagree on values that later tooling has to compare, and the cursor for a snapshot window resolved from one store misfilters the window walked in the other. Three defects in this work passed the full existing test suites before being found by review rather than by a test: the decorator wrote no wait cycle at all, the snapshot window dropped the ordering used to give each completed waitpoint its position in a batch, and the two stores stamped different creation times. The common cause was that no test drove a snapshot that actually carried waitpoints, and that the parity suite compared a timestamp against a value it had just read back from the row it was checking. Both gaps now have tests.
…code, and follow-ups (#4784) Three bugs on the project integrations page, one commit each for the two reported ones and four for the follow-ups found while fixing them. ## `chore`: remove unreachable code on the integrations page (TRI-12645) Two notification panels in `VercelSettingsPanel` could never render: 1. The **"Failed to load Vercel settings"** panel was gated on a `hasError` state whose setter is never called anywhere, so it was permanently `false`. 2. The **"connection expired"** banner *inside* the `connectedProject` branch was unreachable: `VercelSettingsPresenter` only populates `connectedProject` on its success exit, which hardcodes `authInvalid: false`, while both `authInvalid: true` exits return `connectedProject: undefined`. Removing them makes the surrounding `!showAuthInvalid` guards vacuous, and the `onboardingData?.authInvalid` disjunct redundant — the loader already folds onboarding auth state into `authInvalid` before it reaches the component. **No behaviour change.** An org with a connected project and an expired token still gets the banner, from the branch below (untouched). ## `fix`: gate Staging settings on plans without a Staging environment (TRI-12646) The ticket's premise was inverted, and I've corrected it there. In Git settings, **Preview** is the row that's correctly gated; **Staging** is the one with no gate at all: - Preview swaps its switch for an Upgrade button, and `projectSettings.server.ts` neutralises a forged `previewDeploymentsEnabled=on`. - Staging was a plain always-editable `Input`, and `validateStagingBranch` only checked the branch existed on GitHub. An org without a staging environment could type a tracking branch, hit Save, get a success toast, and have it silently do nothing. Staging and Preview environments are created together for projects on a plan that includes them, so gating one and not the other was an oversight. The Staging row now mirrors the Preview row. Server-side it ignores the submitted branch when there's no staging environment, but **preserves the stored branch rather than clearing it** — deliberately different from the Preview handling. Forcing a boolean off is harmless; forcing a *string* off would wipe a tracking branch the org had already configured the first time they saved after losing the environment. The Vercel write path had the same gap: `update-config` / `complete-onboarding` / `update-env-mapping` never re-derived available env slugs server-side, so `["stg","preview"]` could be persisted for a project with neither environment, and `createDefaultVercelIntegrationData` turned preview on unconditionally. Both now filter against the project's actual environments, via a pure `restrictConfigToAvailableEnvSlugs` helper that only touches keys present on the input. ## `fix`: show build settings when the GitHub app is disabled (TRI-13488) The page wrapped Git settings, the Vercel section **and** build settings in one `githubAppEnabled` guard, so with the GitHub app off it rendered an empty container. The Vercel section genuinely depends on GitHub — it can't sync environment variables or link deployments without a connected repo — so it stays gated. Build settings don't: they also apply to CLI deploys run with `--native-build-server`, exactly as the section's own description states. They now render regardless. ## `fix`: stop the Vercel onboarding modal spinning forever (TRI-13488) `computeInitialState` starts in `loading-projects` whenever the org has a Vercel integration but no onboarding data yet, and the effect that escapes it waits for `availableProjects !== undefined`. When `getOnboardingData` returns `null` — it does that on any thrown error, and when the org integration row is missing — nothing ever arrives. The empty-array case self-resolves (`[] !== undefined`), so this is specifically the null case. The route can tell "still loading" from "loaded nothing" because its fetcher always requests `?vercelOnboarding=true`; it now passes that down and the modal explains the failure with a retry and a link to check the integration's access on Vercel. ## `fix`: match staging and preview environments consistently (TRI-13488) The four places that ask "does this project have a staging / preview environment?" disagreed. `VercelSettingsPresenter` matched on type with no parent filter, so any preview *branch* row satisfied it — branches are `PREVIEW` rows too. `GitHubSettingsPresenter` and `ProjectSettingsService` matched on slug instead. Slug is the weaker key: it's derived at creation time and legacy rows can carry something else, which is why `memberDevelopmentEnvironmentWhere` deliberately avoids it. All four now match on `type` plus `parentEnvironmentId: null`, which excludes branches without depending on the slug being canonical. ## `fix`: explain when no Vercel environment can be mapped to Staging (TRI-13488) Reported while reviewing the branch. The Staging build settings show *"Set a Vercel environment for Staging first."* whenever the project has a staging environment and no mapping — but the control that sets the mapping only rendered when the Vercel project had at least one custom environment: ``` hint: hasStagingEnvironment && !configValues.vercelStagingEnvironment control: hasStagingEnvironment && customEnvironments.length > 0 ``` So a Vercel project with no custom environments, or one whose custom environments failed to fetch (the presenter swallows that error to `[]`), got an instruction with nothing to act on. Both conditions predate this PR. The mapping row now always renders alongside the hint and explains what to do when there's nothing to choose from, and the build-settings hint says the same thing. ## `chore`: remove the remaining dead code (TRI-13488) - The `"installing"` `OnboardingState` is unproducible — no `setState` call yields it — so its redirect effect, switch arm, `isLoadingState` conjunct and the `vercelAppInstallPath` import it was the only user of are all dead. - `(state as string) !== "completed"` sits in a branch where TypeScript has already narrowed `"completed"` out; the cast is what let it compile. - `hideSectionToggles` was only ever passed alongside `layout="settings"` but only read inside `layout="card"` blocks, so it could never take effect. Removed the prop entirely. - Unused bindings and the helpers only they referenced: `envSlugLabel`, `_formatSelectedEnvs`, `_CompleteOnboardingForm`, `_handleFinishOnboarding`, and the rest. No behaviour change in that commit. ## Not included The three overlapping modal-open effects in `settings.integrations/route.tsx` are left alone — they're defensive against a close-then-reopen race, and untangling them is a behavioural risk with no user-visible payoff. ## Verification `pnpm run typecheck --filter webapp`, `pnpm run lint` and `pnpm run knip` are clean. New `apps/webapp/test/vercelIntegrationConfig.test.ts` covers the slug restriction and the default-config seeding (both pure functions); 39 tests pass across it and the three existing Vercel/project-settings files. The new `projectId` + `slug` query is served by the existing `@@unique([projectId, slug, orgMemberId])` prefix — same access pattern as the preview check it mirrors. refs TRI-12645, TRI-12646, TRI-13488
…ncy (#4781) Gives read-through and idempotency their gen-2 shard arms, so an id that names its own shard is read there and nowhere else. #4764 has landed, so this now targets `main` directly and no longer depends on an unmerged branch. It builds on what that PR supplied: `resolveShard`, `runOpsShardHandles` and the keyed router. TRI-13431 ## What changes **Read-through routes by `resolveShard`, not by the binary residency classifier.** A gen-2 id reads its own shard's replica once and probes no other store. A gen-1 v1 id still reads new only. **Callers now declare `idKind`.** A cuid gives no way to tell a run id from a waitpoint id, and the two must route differently: - a legacy-classified **run** id reads the legacy replica only — there is no cuid run migration, so the new-store probe cannot find it; - a cuid **waitpoint** keeps the new-first pair probe, which is load-bearing because a cuid waitpoint can be co-located with its run on the new store. There is no default, because a default would pick one of those arms silently. The field `runId` is renamed to `id`, since it carried both kinds already. **`ReadThroughResult` carries `found`.** `source` is an open-ended union once shards exist, so a consumer testing found-ness by listing the hit sources reads a gen-2 hit as a miss. One consumer did exactly that. Discriminating on `found` makes that class of bug a compile error rather than something a reviewer has to spot. **Idempotency resolves its client through one shard-keyed map.** Both call sites go through `clientForShardKey`, so they cannot disagree about which store owns an id. An absent key takes an explicit logged branch to the fallback, not a silent legacy default. The `classify` seam is retyped to return a `ShardKey`: `Residency` (`"NEW"`) and the reserved shard keys (`"new"`) differ only by case, and `ShardKey` collapses to `string`, so the compiler would not have caught feeding one into the other. The dead `isMigrated` branch is deleted. Nothing implemented it, and the one production comment recorded that omitting it was deliberate. **`PostgresRunStore._residency` widens to `ShardKey`.** Still unused; the store stays unaware of its siblings. ## Two behaviour fixes found while doing the above **An unconfigured shard key logs and returns not-found instead of throwing.** The waitpoint route takes the id from a URL parameter, and any base32hex core plus `[a-z0-9]` plus `"2"` parses as gen-2. The route turns a throw into a 500, so throwing here would let any authenticated client generate 500s and error logs by guessing shard chars, of which there are 36. An error-logged not-found is neither silent nor a misroute. Throwing stays correct on the router path, where ids are minted rather than received. **The two cross-seam batch hydration sites were gen-2 blind.** `hydrateRunsAcrossSeam` and `ApiBatchResultsPresenter` classified with the binary `ownerEngine`, so a gen-2 run id joined the gen-1 `new` group, missed there, and — classifying dedicated-family — never reached the legacy probe either. The id was dropped from a bulk-action page and from batch results with no error. Both now partition ids by shard key and read each configured shard once. Also: a gen-2 waitpoint that missed its shard replica fell back to the gen-1 new writer, a different database, silently disabling read-your-writes for the freshly minted token that fallback exists to serve. It now falls back to its own shard's writer. ## Merge safety Inert while `RUN_OPS_SHARDS` is unset: the shard maps are empty, so every gen-2 arm is unreachable, and gen-2 minting is not live yet. The one live change is the gen-1 run arm, and it removes work rather than adding it. `RoutingRunStore.findRun` never forwards the caller's client object — it routes by id and reads only the client's presence and replica brand — so `readRunForEvent`'s "new" closure already resolved a legacy-classified run id to the legacy store. The arm removes a duplicated read of the legacy replica. A test pins this, because a future caller passing a raw client and a run id would lose the pre-cutover 27-char case, which is new-resident but classifies legacy. ## Testing 14 tests added, testcontainers throughout, no mocks. 22 affected test files pass; typecheck, lint, format and knip are clean. Both arms were verified by neutralising them and confirming the new tests fail. The batch-results test needed rewriting after that check: the first version passed with the fix neutralised, because it used one container as both the gen-1 new client and the shard replica, so it was not testing what it claimed. Note for review: run testcontainer suites in small batches. Sixteen at once starves Docker and everything times out at 60 seconds. The run-ops legacy-guard baseline is refreshed in its own commit. The baseline is keyed by line number, so partitioning the batch-results read shifted four pre-existing entries and added one. Baselined violations in that file go from four to five, all reads; the new one is the shard read beside two gen-1 reads already there. No changeset and no `.server-changes` entry: a user notices nothing while the flag is unset.
## ✅ Checklist - [x] I have followed every step in the [contributing guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md) - [x] The PR title follows the convention. - [x] I ran and tested the code works --- ## Testing Reproduced with `useTriggerChatTransport` + `useChat` and the stop pattern from the ai-chat frontend docs: 1. Send a message so a turn is streaming. 2. Call `transport.stopGeneration(chatId)`, then `useChat`'s `stop()`. 3. Send another message. Before this change the second turn never renders: no parts arrive, `status` stays `streaming`, and the session stays `isStreaming: true`, so a stop button stays on screen until the page is reloaded. The run itself is fine and everything persists, so a reload shows the full response. Cause: `stopGeneration` sets `state.skipToTurnComplete = true`, and the read loop only clears that when it sees a `TURN_COMPLETE` record. The abort closes the reader before that record arrives, so the flag survives into the next turn and every record of that turn is skipped, including its own `TURN_COMPLETE`. After this change the same sequence streams the second turn normally. Verified against 4.5.11 and 4.5.12 (both affected) with the equivalent patch applied to the built SDK. --- ## Changelog Reset `skipToTurnComplete` when a new chat turn or action is sent, so a message sent after `stopGeneration` streams normally instead of leaving the chat stuck in a streaming state. --------- Co-authored-by: Devin AI <devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
#4644) ## Summary Adds `chat.messages.hasPending()` and `chat.messages.next()` so a custom agent loop can inspect pending chat input without consuming it and take one record at a time, and fixes four ways a chat could mishandle input across a restart: a message silently lost, a recovered answer cut off by a stop the user had already pressed, a retried send answered twice, and a record the agent had no consumer for blocking every message queued behind it. ```ts if (await chat.messages.hasPending()) { const record = await chat.messages.next({ timeoutInSeconds: 0 }); if (record) handle(record.payload); } ``` ## Why the fixes came together `session.in` carries records for consumers whose delivery needs differ. A user message must be delivered eventually, so it can wait arbitrarily long for a turn to take it. A stop only means anything to the turn that is live when it lands. Progress along the channel was tracked as one sequence number, and one number cannot say "control applied through 7, message 3 still owed" at the same time. Each of the bugs above is that mismatch surfacing somewhere different. So instead of a rule per symptom, records are now classified once and handed to one route, and each route declares two things: whether it holds a record when no consumer is ready, and whether a record it never handled has to survive into the next boot. The resume cursor, the replay window and the discard-the-unowned behaviour are then derived from route state rather than maintained beside it, and `hasPending()` answers from the message queue instead of the head of a buffer shared with every other kind. The wire is unchanged. Both cursors on the turn boundary keep their meanings, so existing chats resume as before and there is no webapp change. ## Behaviour worth calling out `chat.writeTurnComplete()`'s `sessionInEventId` is the cursor that is safe to resume from, not the sequence of the record the turn answered. It is held back behind any message still waiting to be handled, so a value below the record you just handled is expected rather than a sign of a lost turn. The stop fix also covers chats whose most recent turn was completed by an older SDK, by resolving the replay window from the channel when the boundary does not carry one. The trade there is deliberate: a stop that landed in the moments before boot and was never applied is dropped along with the replayed ones, because a stop the user can press again beats a stale one killing an answer they are waiting for. ## Verification Thirteen reproductions against a local stack, each driving real runs rather than mocks, covering the documented `next()`/`hasPending()` loop, suspend and resume, a crash between consuming a message and writing turn-complete, a retried send whose idempotency claim is lost, and a continuation boot that must not replay answered messages. Where applicable each was also run against `main`, so the fixes are differences rather than assertions. Five further legs on a deployed environment, which the earlier revisions of this branch did not cover at all: a message appended while the run is genuinely checkpointed, a message appended while the run is dead, the stop-after-crash case on the real crash path, and both version-skew directions (a newer worker resuming an older worker's turn boundary, and an older worker resuming a newer one's). Two of those restart fixes also have a browser-driven red and green pair on a deployed environment, staged identically on both sides and differing only in the SDK. For the lost-message fix, the unanswered message is replayed and answered in full here, and is never replayed at all on the released SDK. For the stop fix, both sides replay the message and diverge on the stop itself: it is declined here and the answer completes, while the released SDK re-applies it and the recovered answer dies before it streams. The routing decision itself is a pure state machine, so it also has a property test over every interleaving of the record kinds crossed with each crash point, checked by mutation to confirm it fails when the cursor arithmetic or the replay window is broken. ## Known and not addressed here The read of the woken record is unbounded, so a wake with nothing to read makes `wait()` outlive its own waitpoint. Tested and not a deadlock, since the read defers to the next record, but bounding it is a separate change with its own test. Separately, and not caused by this branch: a run that crashes while a message is still queued is not replaced until the next inbound append, so that message waits rather than being recovered on its own. Worth its own issue. Also not caused by this branch, but worth knowing when reading the release note: a chat page that stayed open across the crash keeps showing the partial answer it already received, so the recovered answer only appears after a reload. The answer itself is persisted correctly. The gap is on the client, which does not apply a re-delivered turn over a partial it already holds. --------- Co-authored-by: Eric Allam <eric@trigger.dev> Co-authored-by: Eric Allam <eallam@icloud.com>
## Summary Reloading a browser chat mid-turn can replay a completion event for an older input and close the active turn too early. This persists the last browser-owned input sequence and reuses it on reconnect, so older completion events are ignored. The sequence is cleared after the matching boundary, and reconnect avoids the settled-peek shortcut while that sequence is active. The persisted field is optional, so sessions without it keep their existing behavior. ## Testing - `pnpm --dir packages/trigger-sdk run test ./src/v3/chat.test.ts ./test/chat-turn-correlation.test.ts --run` — 67 passed - `pnpm --dir packages/trigger-sdk run test --run` — 32 files, 379 tests passed - `pnpm run build --filter @trigger.dev/sdk` - `pnpm run format` - `pnpm run lint` ## Changelog Browser chats now keep the active turn open across page reloads when older completion records are replayed. ## ✅ Checklist - [x] I have followed every step in the [contributing guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md) - [x] The PR title follows the convention. - [x] I ran and tested the code works 💯 Co-authored-by: Matt Aitken <matt@mattaitken.com>
Follow-up to [#4644](#4644), now rebased onto main so the diff is just these three commits. ## Summary Two ways a chat could lose a user message, both pre-existing and both raised while reviewing #4644. A message arriving while a turn was streaming was handed to that turn's push handler and parked in an in-memory array. The router counts a record handed to a handler as terminally decided, so it stopped holding the resume floor behind it, and the turn boundary published a cursor past a message that existed only in that process. A crash before the next turn lost it, silently. Measured: with the message at sequence 1, the boundary published `session-in-event-id: 1`, so a resume skipped it. Separately, a message the agent declined to inject was discarded with the turn. Never injected, never written to the wire buffer, never answered. That was also the documented default, since a `pendingMessages` config without `shouldInject` declines every batch. ## Design Notification and consumption are now separate concerns on the router. `observe` reports that a record arrived without taking it, so the record stays queued and keeps holding the floor. It is rejected on an `at-arrival` route: an observer there would either have to count as a listener, which would stop an unconsumed stop being discarded and bring back a wedged mailbox, or watch records it cannot affect. `take` removes exactly one queued record. The managed loop and the `chat.createSession()` iterator now only subscribe when there is a steering config to feed, and injection is the point of consumption. A declined batch never reaches the take, so its records stay queued and become later turns. Both in-memory wire buffers are gone, so a message waiting for its turn is durable rather than living in whichever worker received it. The floor doubles as the wake cursor: `awaitWake` registers with it and the server completes the waitpoint immediately if anything sits after that sequence. An over-advanced floor was therefore also a missed wake. It is now recorded on the wait span so a run that never woke can be diagnosed from its trace. ## Verification Both fixes have a red and green pair, each checked against the unmodified source rather than only observed to pass: - the resume cursor test fails on the parent branch and passes here - the declined-message test fails without the second commit and passes with it Also 8 new router tests for `observe` and `take`. Suites green at 385 for the SDK and 886 for core. ## Not addressed A `pendingMessages` config with no `chat.toStreamTextOptions()` spread still swallows messages, because nothing drains the queue at all. Same shape, different trigger, tracked separately.
## Summary Raw `chat.customAgent()` loops can now call `chat.endAndContinue()` to move the Session to a fresh run. The managed loop already used the same server operation through `chat.requestUpgrade()`, but raw loops could not call it directly. Call the method between turns after detaching input listeners from the old run. Await it and return immediately. Unconsumed `.in` records stay on the Session for the continuation run. I put this on the `chat` namespace next to the other raw chat primitives. Happy to move it if maintainers prefer a different API placement. ## Testing - `pnpm exec vitest run` in `packages/trigger-sdk` (374 tests) - Focused webapp Session E2E tests (3 tests) - `pnpm run build` in `packages/trigger-sdk` - Webapp typecheck - `pnpm run format` - `pnpm run lint` ## Checklist - [x] I followed the contributing guide - [x] The PR title follows the convention - [x] I tested the change ## Changelog Allow custom chat agents to rotate to a new task version without dropping unconsumed Session input. --------- Co-authored-by: Eric Allam <eallam@icloud.com>
…aitpoint ids (#4788) ## Summary Adds the id-minting half of sharding run data across several databases. Every entity that co-locates with a run now carries the run's shard key inside its own id, so its row is routable on its own instead of needing a directory table or a scatter across shards. Nothing changes for users yet. With no shard descriptors configured, every mint path produces exactly the ids it produces today, and the trigger path issues no extra query. ## Design A run's mint target travels as a single object carrying the kind and, when sharded, the shard character. The shard and the caller's region both occupy index 24 of a run-ops id, so passing them together makes it impossible for a caller to set two competing sources for one slot. A child run, a batch and a batch item read the shard from their parent's id rather than resolving a fresh one, so a run tree never splits across databases. Three services carried that branch separately, and one had already drifted, so it now lives in one function. Waitpoints mint through one shared pure function used by both the webapp and the run engine. They have to agree byte for byte, because the routing store refuses a waitpoint whose id is not stamped for the shard it is being written to: ```ts mintWaitpointIdForShard(key) // standalone token: the environment's shard mintWaitpointIdFor(anchorId) // co-located: the anchor's shard, or a cuid ``` The core is always freshly minted rather than derived from the anchor, since a derived body would be byte-identical to the run's own id. One latent bug fixed on the way: the failed-run path duplicated the mint branch inline and had drifted, so a child of a sharded parent would have been written to a different database from its parent. ## Guarding the create sites The expensive failure here is a waitpoint minted without its anchor's shard: one of the five create sites writes through a path that has no stamp check, so a miss there strands a blocked run with nothing logged. An enumerated census plus a source scan fails when a new create site appears, when an existing one stops passing its anchor, or when a site is added to a file the scan does not yet cover. The census was written before any site was converted, so it went red on the first commit and green as the last site landed. Both holes an earlier draft had, a file-granular count and a scan that missed the directory these mints used to live in, were confirmed closed by reintroducing them and watching the guard fail. ## Before enabling a shard Merging this is inert: with the mint list empty the resolver returns before it reads anything, and ids are identical to a measured `main` baseline. Verified against a live shard locally, including that the resolver issues no query across thirty triggers with no shard configured. Enabling is gated on two other pull requests, both open, both by the same author, each of which owns the file involved: - **#4781** adds the gen-2 shard arm to read-through. Without it a gen-2 run cannot wait on a token at all: the wait route resolves the waitpoint through read-through, which is shard-blind, so the wait fails. Do not set the mint list before it merges. - **#4780** generalises the distinct-database sentinel. Without it a shard pointed at the same physical database as the gen-1 store boots without complaint, which voids the disjointness the fan-out sums rely on. Testing also turned up a silent read-path gap that neither pull request covers: the paths that hydrate runs from ClickHouse through a fixed pair of Postgres clients drop gen-2 rows on the floor, so the runs list would show fewer rows than its own count with nothing logged. That needs its own change before a shard carries real traffic, and it is filed as such. ## Notes for reviewers Four commits in the middle of the stack do not typecheck in isolation: a signature change and its call-site repairs are separate commits, so bisecting inside the stack needs care. Commit `845ab06` also understates itself, since it rewrites the primary trigger path's mint alongside the failed-run path it names. No changeset and no server-changes entry: every path is inert while the feature is off, so there is nothing to tell users yet. --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
<!-- ccr-slack-attribution --> _Requested by **Matt Aitken** · [Slack thread](https://triggerdotdev.slack.com/archives/C061L2MHW93/p1787615162456839?thread_ts=1787615162.456839&cid=C061L2MHW93)_ **Before:** a `chat.agent` run is killed mid-answer (OOM, crash, eviction) while the message it was answering is the only one still outstanding. The new run boots, puts that message and the half-written reply into its context, and then waits for a message that already arrived. Nobody ever answers the user; the run sits idle until it times out. **After:** the new run re-runs that message as a fresh turn and replies to it. The half-written reply is dropped. When two or more messages are outstanding, nothing changes — the interrupted one still goes into context and the newer ones are re-run, exactly as before. ## ✅ Checklist - [x] I have followed every step in the [contributing guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md) - [x] The PR title follows the convention. - [x] I ran and tested the code works --- ## Testing New regression test in `packages/trigger-sdk/test/recovery-boot.test.ts` — seeds a partial assistant plus exactly one in-flight user, no `onRecoveryBoot`, and asserts one turn fires for that user with the orphan partial dropped from the chain. It fails on `main` (`turnCount` 0, no turn at all) and passes with this change. - `pnpm exec vitest run` in `packages/trigger-sdk` — 373 passed, 1 skipped (31 files passed, 1 skipped) - `pnpm exec oxfmt --check` on the changed files — clean - `pnpm exec oxlint packages/trigger-sdk/src packages/trigger-sdk/test` — clean - `pnpm run build --filter @trigger.dev/sdk` — clean **What it does:** with exactly one in-flight user on a recovery boot, re-dispatch that user as a fresh turn instead of splicing it into the seed chain, where it was never answered. **How:** the recovery-boot smart default made one decision in two halves — the seed chain and the recovered-turn list — both gated on `partialAssistant !== undefined && inFlightUsers.length > 0`. The splice consumes `inFlightUsers[0]` into the chain as "the question the partial was answering" and dispatches the rest. That only works when there *is* a rest: at n=1 `recoveredTurns` came out empty, the boot-injected queue stayed empty, the `session.in` cursor was advanced past the message anyway, and on a `preload` or continuation boot (no `message` on the wire payload) neither dispatch site fired. Both branches now require `length > 1`, so n=1 falls through to the documented default — chain = `settledMessages`, re-dispatch every in-flight user. The submit-message boot is unaffected: the existing dedup still drops a queued message identical to the one already on the wire payload. Also corrected alongside it: the two SDK docstrings and the `docs/ai-chat/patterns/recovery-boot.mdx` defaults section, which described the default as "re-dispatch every user" and never mentioned the splice. Follow-up (not in this PR): the webapp e2e OOM helper never streams a token before throwing, so it exercises the no-partial path only and would not have caught this. Worth a variant that emits a token first. --- ## Changelog Fixed a chat agent hanging after an interrupted turn: when a run was killed mid-answer and only the one message it was answering was still outstanding, the new run never replied to it. That message is now re-answered on the new run. Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Matt Aitken <matt@mattaitken.com> Co-authored-by: Eric Allam <eric@trigger.dev>
## Summary
`chat.withClientData({ schema }).customAgent()` now parses
`payload.metadata` before passing it to `run`, `chat.messages`, or
`chat.createSession`. Schema defaults and transforms are preserved.
Custom agents without a schema keep the existing pass-through behavior.
This does not change `chat.agent()`. Raw custom agents do not expose an
action schema, so `payload.action` remains `unknown`.
## Validation failures
Invalid client data is logged and never passed to user code. The client
receives a fixed `Invalid client data` error; validator details stay in
the task log and `onClientDataValidationError`.
- Submitted turns and async reads write the error followed by
`turn-complete`, then wait for the next valid frame. This settles the
invalid input before the raw read returns. Callers that need to
coordinate validation with their own persistence or settlement should
omit the schema and validate the full frame in their loop.
- Messageless preload and continuation boots call
`onClientDataValidationError` and wait without writing a terminal frame.
- Active `chat.messages.on()` subscriptions skip invalid frames and call
`onClientDataValidationError` without ending the response. `off()` stops
new frames. A valid frame accepted before `off()` finishes validation
and is delivered; an invalid pending frame is logged without invoking
user callbacks.
- `chat.messages.peek()` throws synchronously.
- Invalid head-start handovers fail closed. A skip ends the run. A real
handover writes the validation error after the warm output, writes
`turn-complete`, and ends the run.
Validation is automatic when a schema is declared. We can make it opt-in
or return a typed failure if maintainers prefer that contract.
## Testing
- `pnpm --filter @trigger.dev/sdk run test -- --run`
- `pnpm --filter @trigger.dev/sdk run typecheck`
- `pnpm run build --filter @trigger.dev/sdk`
- `pnpm run lint`
- Formatting checks pass
## ✅ Checklist
- [x] I followed the contributing guide
- [x] The PR title follows the convention
- [x] I ran and tested the change
## Changelog
Custom chat agents now validate and parse client data declared with
`chat.withClientData({ schema })` before passing it to agent code.
## Screenshots
Not applicable.
---------
Co-authored-by: Eric Allam <eallam@icloud.com>
The CLI now asks the server which build path to use before it builds or
uploads anything, so native builds can be rolled out per organization
and per environment type without a CLI release.
```
trigger.dev deploy
│
├─ explicit flag? (--native-build / --local-build / --depot-build)
│ └─ yes → use it, never ask the server
│
└─ GET /api/v1/projects/:ref/:env/deploy-settings (env API key, 5s timeout, one attempt)
│
│ server resolves: native unavailable → org[env type] → org → global[env type] → global → depot
│
├─ { "build_path": "native" | "native_local_bundle" } → that path
├─ { "build_path": "depot" } → Depot
└─ error / timeout / 404 → Depot (fail open)
```
The path comes from four enum feature flags, editable in the global and
per-org admin flag UIs: `deployBuildPath` and `deployBuildPathPreview` /
`Staging` / `Production`. Unset everywhere keeps current behaviour
unchanged; CLIs older than this release never call the endpoint and keep
their current behaviour.
## Summary 4 new features, 12 improvements, 5 bug fixes. ## Improvements - `trigger.dev deploy` now asks the server whether to build with Depot or the native build server unless `--native-build`, `--depot-build`, or `--local-build` is passed, so the native build server can be rolled out per organization without a CLI change. `--local-bundle` and `--detach` now require `--native-build`. ([#4803](#4803)) - Add an experimental `--local-bundle` deploy flag that runs the install and bundling steps on your machine and uploads only the build output; the image is still built remotely. Useful when your project's install step needs tooling or credentials that only exist locally. ([#4331](#4331)) - Send the CLI version header on all API requests so deployments are attributable to a CLI version ([#4778](#4778)) - A message that arrives mid-turn and is not injected into that turn is now answered as the next turn, instead of being dropped. This is what the `pendingMessages` docs have always described, and it applies to the default too: configuring `pendingMessages` without a `shouldInject` declines every batch, which previously meant every mid-turn message was lost with no error at either end. ([#4795](#4795)) ```ts chat.agent({ id: "my-chat", pendingMessages: { onReceived: ({ message }) => logger.info("arrived mid-turn", { id: message.id }), // Only interrupt once the agent has started calling tools. shouldInject: ({ steps }) => steps.length > 0, }, run: async ({ messages, signal }) => streamText({ model, messages, abortSignal: signal, // Required for injection. Without it nothing injects, and every // mid-turn message is answered as the next turn instead. ...chat.toStreamTextOptions(), }), }); ``` A declined message keeps its place in the queue, so it survives a crash and is answered by whichever run picks the conversation up. An injected one is consumed at the moment it is injected, so it is never also answered as a later turn. - Browser chats now keep the active turn open across page reloads when older completion records are replayed. ([#4643](#4643)) - Add `chat.endAndContinue()` so fully hand-rolled custom chat agents can hand a conversation off to a fresh run on the latest deployed task version while preserving unconsumed Session input. ([#4647](#4647)) - Custom chat agents now validate and parse client data declared with `chat.withClientData({ schema })` before passing it to agent code. ([#4646](#4646)) ## Bug fixes - Fixes a case where a chat could silently lose a message. If a message arrived while the agent was between turns and a stop arrived after it, the cursor the next boot resumed from could point past that message, so it was never answered and no error was raised. This affected `chat.agent`, not just custom agents. ([#4644](#4644)) Fixes a recovered answer being cut off. After a crash the agent replays the message it had not answered yet, but it was replaying the stop that arrived after that message too, so the turn answering it was aborted the moment it began. A stop is now only applied to the turn that was live when it arrived. That holds however the stop got there: sent after the last completed turn, or sent to a chat whose most recent turn was completed by an older version of the SDK. One limitation to know about: the recovered answer is persisted correctly, but a chat page that stayed open across the crash keeps showing the partial answer it had already received. Reload the page to see the full recovered answer. Also fixes a retried send being answered twice. When a send was retried and its idempotency claim was lost, the agent could consume the same message a second time. Custom agent loops can now inspect pending chat input without consuming it, and consume one record at a time, with `chat.messages.hasPending()` and `chat.messages.next()`. Records carry stable identifiers so a redelivery is recognisable. ```ts if (await chat.messages.hasPending()) { const record = await chat.messages.next({ timeoutInSeconds: 0 }); if (record) handle(record.payload); } ``` `hasPending()` answers for messages alone, so a message sitting behind a stop, or behind a record this version of the SDK does not recognise, still reports as pending and is still delivered. Anything the agent has no consumer for is discarded rather than left where it would make every message queued behind it undeliverable. `chat.messages.next()` returning `undefined` means no message became consumable before the timeout. `chat.writeTurnComplete()`'s `sessionInEventId` is the cursor that is safe to resume from, not the sequence of the record the turn answered. It is held back behind any message still waiting to be handled, so a value below the record you just handled is expected. - Fixed a chat agent hanging after an interrupted turn: when a run was killed mid-answer (out of memory, crash, or eviction) and only the one message it was answering was still outstanding, the new run never replied to it. That message is now re-answered on the new run. ([#4768](#4768)) - Fix chat transport discarding the next turn after stopping generation. `skipToTurnComplete` is now reset when a new message or action is sent, so a message sent after `stopGeneration` streams normally instead of leaving the chat stuck in a streaming state. ([#4744](#4744)) - Fixes a message sent while the agent was mid-answer being lost if the run then crashed. The cursor written at the end of each turn could point past a message that had arrived during that turn but had not been answered yet, so the next boot skipped it and no error was raised anywhere. Such a message is now held until a turn actually takes it. ([#4795](#4795)) This also removes the in-memory buffer those messages used to sit in, on both `chat.agent` and `chat.createSession()`, so a message waiting for its turn is durable rather than only present in the worker that received it. ## Server changes These changes affect the self-hosted Docker image and Trigger.dev Cloud: - Self-hosted instances can now disable the admin dashboard and user impersonation entirely. See the self-hosting docs for the new setting. ([#4774](#4774)) - The dashboard has two new themes, Black and White, plus appearance options for stronger colors and underlined links. ([#4547](#4547)) - Deployment logs no longer jump to the bottom while you are reading earlier output. Scroll up to pause auto-scroll, and scroll back down or use the new scroll-to-bottom button in the log header to resume following. ([#4776](#4776)) - Customize the runs list: show, hide, and reorder columns, and add smart columns that pull a value straight out of a run's payload, metadata, or output. Your column choices are saved in the page URL, so you can share a view, bookmark it, or save it straight to your favorites. ([#4652](#4652)) - Stop the browser offering to autofill or save environment variable values as saved credentials. ([#4777](#4777)) - Cut webapp CPU usage by about a quarter on the routes that workers call most, freeing headroom at the same request rate. Detailed event-loop blocking traces are no longer recorded by default, because producing them was itself a large part of that cost. ([#4746](#4746)) - When a runs list or runs.list API request spans too much data to complete, it now returns a clear, actionable error asking you to narrow the time range, instead of failing with a generic error. ([#4773](#4773)) - Improved the performance and reliability of the runs list and the runs.list API, especially for large projects and filtered views. ([#4763](#4763)) - New Vercel connections now get version skew protection turned on automatically, so each run uses the task version its deployment shipped with. Automatic atomic deployments are deprecated and no longer offered when you connect a project, but stay available in your Vercel integration settings. ([#4741](#4741)) - The Staging branch setting now shows an upgrade prompt on plans that don't include a Staging environment, instead of looking editable and then silently doing nothing when saved. ([#4784](#4784)) <details> <summary>Raw changeset output</summary> # Releases ## @trigger.dev/build@4.5.13 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.13` ## trigger.dev@4.5.13 ### Patch Changes - `trigger.dev deploy` now asks the server whether to build with Depot or the native build server unless `--native-build`, `--depot-build`, or `--local-build` is passed, so the native build server can be rolled out per organization without a CLI change. `--local-bundle` and `--detach` now require `--native-build`. ([#4803](#4803)) - Add an experimental `--local-bundle` deploy flag that runs the install and bundling steps on your machine and uploads only the build output; the image is still built remotely. Useful when your project's install step needs tooling or credentials that only exist locally. ([#4331](#4331)) - Send the CLI version header on all API requests so deployments are attributable to a CLI version ([#4778](#4778)) - Updated dependencies: - `@trigger.dev/core@4.5.13` - `@trigger.dev/build@4.5.13` - `@trigger.dev/schema-to-json@4.5.13` ## @trigger.dev/core@4.5.13 ### Patch Changes - `trigger.dev deploy` now asks the server whether to build with Depot or the native build server unless `--native-build`, `--depot-build`, or `--local-build` is passed, so the native build server can be rolled out per organization without a CLI change. `--local-bundle` and `--detach` now require `--native-build`. ([#4803](#4803)) - Add an experimental `--local-bundle` deploy flag that runs the install and bundling steps on your machine and uploads only the build output; the image is still built remotely. Useful when your project's install step needs tooling or credentials that only exist locally. ([#4331](#4331)) - A message that arrives mid-turn and is not injected into that turn is now answered as the next turn, instead of being dropped. This is what the `pendingMessages` docs have always described, and it applies to the default too: configuring `pendingMessages` without a `shouldInject` declines every batch, which previously meant every mid-turn message was lost with no error at either end. ([#4795](#4795)) ```ts chat.agent({ id: "my-chat", pendingMessages: { onReceived: ({ message }) => logger.info("arrived mid-turn", { id: message.id }), // Only interrupt once the agent has started calling tools. shouldInject: ({ steps }) => steps.length > 0, }, run: async ({ messages, signal }) => streamText({ model, messages, abortSignal: signal, // Required for injection. Without it nothing injects, and every // mid-turn message is answered as the next turn instead. ...chat.toStreamTextOptions(), }), }); ``` A declined message keeps its place in the queue, so it survives a crash and is answered by whichever run picks the conversation up. An injected one is consumed at the moment it is injected, so it is never also answered as a later turn. - Fixes a case where a chat could silently lose a message. If a message arrived while the agent was between turns and a stop arrived after it, the cursor the next boot resumed from could point past that message, so it was never answered and no error was raised. This affected `chat.agent`, not just custom agents. ([#4644](#4644)) Fixes a recovered answer being cut off. After a crash the agent replays the message it had not answered yet, but it was replaying the stop that arrived after that message too, so the turn answering it was aborted the moment it began. A stop is now only applied to the turn that was live when it arrived. That holds however the stop got there: sent after the last completed turn, or sent to a chat whose most recent turn was completed by an older version of the SDK. One limitation to know about: the recovered answer is persisted correctly, but a chat page that stayed open across the crash keeps showing the partial answer it had already received. Reload the page to see the full recovered answer. Also fixes a retried send being answered twice. When a send was retried and its idempotency claim was lost, the agent could consume the same message a second time. Custom agent loops can now inspect pending chat input without consuming it, and consume one record at a time, with `chat.messages.hasPending()` and `chat.messages.next()`. Records carry stable identifiers so a redelivery is recognisable. ```ts if (await chat.messages.hasPending()) { const record = await chat.messages.next({ timeoutInSeconds: 0 }); if (record) handle(record.payload); } ``` `hasPending()` answers for messages alone, so a message sitting behind a stop, or behind a record this version of the SDK does not recognise, still reports as pending and is still delivered. Anything the agent has no consumer for is discarded rather than left where it would make every message queued behind it undeliverable. `chat.messages.next()` returning `undefined` means no message became consumable before the timeout. `chat.writeTurnComplete()`'s `sessionInEventId` is the cursor that is safe to resume from, not the sequence of the record the turn answered. It is held back behind any message still waiting to be handled, so a value below the record you just handled is expected. ## @trigger.dev/python@4.5.13 ### Patch Changes - Updated dependencies: - `@trigger.dev/sdk@4.5.13` - `@trigger.dev/core@4.5.13` - `@trigger.dev/build@4.5.13` ## @trigger.dev/react-hooks@4.5.13 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.13` ## @trigger.dev/redis-worker@4.5.13 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.13` ## @trigger.dev/rsc@4.5.13 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.13` ## @trigger.dev/schema-to-json@4.5.13 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.13` ## @trigger.dev/sdk@4.5.13 ### Patch Changes - Fixed a chat agent hanging after an interrupted turn: when a run was killed mid-answer (out of memory, crash, or eviction) and only the one message it was answering was still outstanding, the new run never replied to it. That message is now re-answered on the new run. ([#4768](#4768)) - Browser chats now keep the active turn open across page reloads when older completion records are replayed. ([#4643](#4643)) - Add `chat.endAndContinue()` so fully hand-rolled custom chat agents can hand a conversation off to a fresh run on the latest deployed task version while preserving unconsumed Session input. ([#4647](#4647)) - Fix chat transport discarding the next turn after stopping generation. `skipToTurnComplete` is now reset when a new message or action is sent, so a message sent after `stopGeneration` streams normally instead of leaving the chat stuck in a streaming state. ([#4744](#4744)) - Custom chat agents now validate and parse client data declared with `chat.withClientData({ schema })` before passing it to agent code. ([#4646](#4646)) - Fixes a message sent while the agent was mid-answer being lost if the run then crashed. The cursor written at the end of each turn could point past a message that had arrived during that turn but had not been answered yet, so the next boot skipped it and no error was raised anywhere. Such a message is now held until a turn actually takes it. ([#4795](#4795)) This also removes the in-memory buffer those messages used to sit in, on both `chat.agent` and `chat.createSession()`, so a message waiting for its turn is durable rather than only present in the worker that received it. - A message that arrives mid-turn and is not injected into that turn is now answered as the next turn, instead of being dropped. This is what the `pendingMessages` docs have always described, and it applies to the default too: configuring `pendingMessages` without a `shouldInject` declines every batch, which previously meant every mid-turn message was lost with no error at either end. ([#4795](#4795)) ```ts chat.agent({ id: "my-chat", pendingMessages: { onReceived: ({ message }) => logger.info("arrived mid-turn", { id: message.id }), // Only interrupt once the agent has started calling tools. shouldInject: ({ steps }) => steps.length > 0, }, run: async ({ messages, signal }) => streamText({ model, messages, abortSignal: signal, // Required for injection. Without it nothing injects, and every // mid-turn message is answered as the next turn instead. ...chat.toStreamTextOptions(), }), }); ``` A declined message keeps its place in the queue, so it survives a crash and is answered by whichever run picks the conversation up. An injected one is consumed at the moment it is injected, so it is never also answered as a later turn. - Fixes a case where a chat could silently lose a message. If a message arrived while the agent was between turns and a stop arrived after it, the cursor the next boot resumed from could point past that message, so it was never answered and no error was raised. This affected `chat.agent`, not just custom agents. ([#4644](#4644)) Fixes a recovered answer being cut off. After a crash the agent replays the message it had not answered yet, but it was replaying the stop that arrived after that message too, so the turn answering it was aborted the moment it began. A stop is now only applied to the turn that was live when it arrived. That holds however the stop got there: sent after the last completed turn, or sent to a chat whose most recent turn was completed by an older version of the SDK. One limitation to know about: the recovered answer is persisted correctly, but a chat page that stayed open across the crash keeps showing the partial answer it had already received. Reload the page to see the full recovered answer. Also fixes a retried send being answered twice. When a send was retried and its idempotency claim was lost, the agent could consume the same message a second time. Custom agent loops can now inspect pending chat input without consuming it, and consume one record at a time, with `chat.messages.hasPending()` and `chat.messages.next()`. Records carry stable identifiers so a redelivery is recognisable. ```ts if (await chat.messages.hasPending()) { const record = await chat.messages.next({ timeoutInSeconds: 0 }); if (record) handle(record.payload); } ``` `hasPending()` answers for messages alone, so a message sitting behind a stop, or behind a record this version of the SDK does not recognise, still reports as pending and is still delivered. Anything the agent has no consumer for is discarded rather than left where it would make every message queued behind it undeliverable. `chat.messages.next()` returning `undefined` means no message became consumable before the timeout. `chat.writeTurnComplete()`'s `sessionInEventId` is the cursor that is safe to resume from, not the sequence of the record the turn answered. It is held back behind any message still waiting to be handled, so a value below the record you just handled is expected. - Updated dependencies: - `@trigger.dev/core@4.5.13` </details> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## Summary Several of the run-ops router's rules only apply above two stores, and the fake-slot suites only ever built two, so those rules were untestable by construction. `clearIdempotencyKey` is the sole caller of the "every other shard" helper, and with two stores that helper returns a single entry, which hides a take-the-first bug. The absent-id partition has the same blind spot: a gen-2 id and a cuid select the same store when only one other store exists. Three suites now run at two shards and at three, with the expected value indexed by topology wherever the rule genuinely changes. The fourth stays at two and says why in the file, because its N-shard behaviour is already pinned in `runOpsStore.shardMap.test.ts`. Two webapp tests defined their own local `RoutingRunStore`. They compiled against a two-store model whatever the real class did, and one described a routing rule the code never implemented. Both now build the real router over the two Postgres stores they already create. ## Validating a test-only change Every new assertion passed the first time it ran, which proves nothing. Each was checked by breaking the router in the way the test claims to guard, then confirming the failure lands in the three-shard arm while the two-shard arm still passes: - take-the-first fan-out in the "every other shard" helper - gen-2 keys moved to the front of the merge precedence order - the absent-id partition sending every id to the gen-1 pair, which fails as `expected +0 to be 1`, the shape a silently under-counted waitpoint takes - residency routing disabled entirely, caught by 3 of the 5 webapp tests Each mutation was reverted. No production code changes. One note for anyone extending these: the webapp resolves `@internal/run-store` to `dist/`, not to source, so a source edit without a rebuild makes those two tests assert against the previous router and pass.
…#4811) ## Summary Realtime streams get a live "last value" mode: subscribe from the latest record instead of replaying the whole history, keep memory bounded, and resume across reloads. Plus a new `useSessionStream` hook for reading a Session's channels from React. ## `useRealtimeStream`: start-from-latest, bounded, resumable ```tsx const { parts, lastEventId } = useRealtimeStream<Frame>(runId, "frames", { from: "latest", // skip history, only new records after connect maxParts: 1, // keep just the most recent (bounded memory) lastEventId: saved, // resume from a persisted cursor (survives reload) onParts: (batch) => save(batch.at(-1)?.id), // per-batch event ids accessToken, }); ``` `from`, `lastEventId` (option and return), and the batching also apply to `streams.read()` and `fetchStream()`. ## `useSessionStream`: read a Session channel from React (new) A read-only hook for a Session's `out` (default) or `in` channel, with the same start / bound / resume options. `useSession` is reserved for two-way (read and write). ```tsx const { records, lastEventId } = useSessionStream<Frame>(sessionId, { io: "out", from: "latest", maxRecords: 5, onRecords: (batch) => {/* each throttled batch, with event ids */}, accessToken, }); ``` ## Access-token refresh Long-lived subscriptions can survive token expiry: pass `refreshAccessToken` and a 401/403 triggers one re-mint and reconnect. With no refresher, auth errors stay terminal exactly as before. ```tsx const { parts } = useRealtimeStream<Frame>(runId, "frames", { accessToken, // called on a 401/403 to mint a fresh public token from your backend refreshAccessToken: async () => { const res = await fetch("/api/realtime-token"); return (await res.json()).token; }, }); ``` It is also available on `useApiClient` / `TriggerAuthContext`, so every hook under a provider shares one refresher. ## Notes Server support (S2 `tail_offset` / Redis `$`, and the start-position header on the run and session SSE routes) ships here; a client passing `from: "latest"` against an older server degrades safely to a full replay. Resume, bounded memory, batched callbacks, and token refresh are client-only. Supersedes #4808 and #4809, folded in here. Verified end to end on an isolated stack: `from: "latest"` on the run and session paths against real S2, `lastEventId` resume across a reload, bounded memory, batched callbacks, and a real 401 to token-refresh to reconnect.
Visual-only changes to the Tasks-page onboarding blank state (brand-new project, dev environment). Formatting, lint, and knip pass via the pre-push hooks; open the Tasks page for a new project to confirm the panel, copy button, and step 2 render as intended. --- ## Changelog Polished the "Set it up with your AI agent" onboarding panel: top-aligned the badge and switched it to the custom Ask AI sparkle icon, stopped the copy-prompt button from resizing when it swaps to "Copied prompt" (the bright check icon now sits beside the label), removed the sparkle from the button's idle state, removed the spinner next to "Start the dev server", and widened the gap between the panel text and the copy button. --- ## Screenshots <img width="800" height="643" alt="CleanShot 2026-08-27 at 19 04 43" src="https://github.com/user-attachments/assets/aede4ca1-30d5-4240-aa18-e1a20161973d" /> 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- conductor-workspace-link --> --- [Open workspace in Conductor](https://app.conductor.build/workspace/d4a21ab5-d6fa-4de2-b5f0-4f34abf0b8b9) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary Batches created on a run-ops store other than the two the list reads were missing from the Batches page. No error, nothing logged: the page just showed fewer batches than exist. This is only reachable once additional run-ops stores are configured, so nothing changes for anyone today. ## Fix The list scanned exactly two databases and merged them by keyset. It now covers one leg per configured store, in ascending precedence order, all issued together. The existing keyset merge generalises without change. Every leg runs the same query, with the same cursor predicate, ordering and over-fetch, so a row's rank within its own leg is never worse than its global rank, and the merged first page is still the true first page. That argument holds for any number of legs, not just two. The empty-state check keeps its existing sequential pair, since a project with no batches is the common case for that path, then issues the remaining checks in a single round trip. A store that declares itself an alias of another shares its client by reference, so it contributes no leg. Scanning it would query the same database twice for rows the other leg already returned. This matches how the routing store and the boot checks treat an alias. The fan-out deliberately fails the page if any store is unreachable, rather than returning a short page. A tolerant merge would recreate the same silent absence this change removes, with a wider blast radius. ## Verification Covered by container tests against real databases: gen-1, legacy and additional stores merged into one ordered page, paging forward and back across a boundary that spans stores, and the empty-state check. Also verified end to end against a live environment with a real corpus: the missing rows reproduce with the new leg removed and appear correctly with it present, ordering interleaves across stores as expected, paging across a store boundary loses and repeats nothing, and the page is byte-identical to before when no extra store is configured. Merge precedence is pinned by its own test: one id seeded on two stores, asserting the higher-authority copy is the one shown. Verified by mutation, since a union-only test passes regardless of leg order. ## Boot interlocks Two related boot checks changed alongside the read path, since configuring an extra store is what makes them reachable. A store configured while split reads are disabled is dropped in silence: no client is built, no leg is added, and rows already resident there disappear from every list with no error. The other two ways the split ends up disabled already refuse to start; this closes the one that did not, and names the stores it is refusing. A store that declares itself an alias of another owns no database, so it is exempt. The distinct-database probe fails closed, which meant one store being briefly unreachable collapsed the deployment to single-DB and then refused the boot entirely. Each target now gets a bounded number of attempts with a short backoff before the probe gives up. Failing closed is unchanged once that budget is exhausted, and a genuine duplicate is still a final answer that is never retried.
Keep steering messages in context across later agent steps and preserve their original position in saved conversations and subsequent turns. Persist managed custom response data in stream order and retain prepared steering input when a conversation resumes in a new worker. Mono-RevId: 0c642ea561d632dabb68ad024aaff4e7e708d114
Mono-RevId: 5997ba1b23b730bd390acfc3ddd3c8c60f06e00a
) ## Summary Follow-up to [#4952](#4952). The new "Side effects inside `run()`" section told readers to derive the provider's idempotency key from a payload ID, but the more general answer is the run ID: `ctx.run.id` is stable across every attempt of a run, is always available, and is the same value the default `run` scope already mixes into a Trigger.dev idempotency key. The Stripe refund example now keys the provider call on `ctx.run.id`, and a short paragraph explains when a business ID from the payload is the better choice (when the same task can be triggered more than once for the same order and you want deduplication across separate runs). Docs only.
Mono-RevId: b32e17efc094e8641bd7e2ccc544a9c9aeae470f
Add a CI guard that checks new Prisma migrations for idempotent, lock-safe DDL: creates must use IF NOT EXISTS, drops IF EXISTS, CREATE TYPE, ADD CONSTRAINT and RENAME must sit in a guarded DO block, INSERTs need ON CONFLICT, and indexes on existing tables must be built CONCURRENTLY in a single-statement migration. The enforced cutoff date is pinned in the `guard:migrations` script in `apps/webapp/package.json`; `-- --all` audits the whole history locally. A `-- migration-guard: allow <reason>` comment opts a single statement out. Mono-RevId: 7c937700bc4eae8b5b51fecdb60b620bf3c9807b
Mono-RevId: c77f5700f7d4a98e7e2a86685f4a7285960e82cd
…erify tokens ## Summary Hosted webhook sources can now describe how the provider expects to be answered, and the ingress honors it. This is the server side for providers like Discord and WhatsApp, whose SDK sources follow in a later release. - A verifier artifact can declare a response contract as data: the status code for a handshake answer (`respondStatus`), and the codes returned for accepted deliveries and rejected signatures (`response.acceptedStatus`, `response.rejectedStatus`). The ingress and the dashboard's send action map every outcome through the same helper, so a source that needs 204 on success and 401 on a bad signature gets exactly that. - A verifier artifact can declare a GET verification flow (`getHandshake`) for providers that confirm a callback URL with a challenge, such as Meta's `hub.challenge`. The ingress answers GET on the endpoint URL against a dedicated verify token, which you generate and reveal from the endpoint's Connect panel. The token is its own credential, separate from the signing secret. - HMAC verifiers can read the signed timestamp from a body field. The Linear provider uses it for a 60 second replay window on `webhookTimestamp`; deliveries are deduplicated on the signed request itself, so a replay with a different unsigned delivery header is still recognised as a duplicate. - The dashboard's test-send re-signs a recorded sample with the current timestamp, so sources with a body-timestamp replay window accept it. - An admin API action bootstraps delivery partitions in the configured webhook database before enablement. It is safe to repeat and preserves existing partitions. Bootstrap and daily maintenance can use `WEBHOOK_DATABASE_DIRECT_URL` with separate owner credentials while application queries use `WEBHOOK_DATABASE_URL`. When the direct URL is unset, partition operations reuse the webhook writer. - The index worker protocol gains a message for duplicate webhook ids, so the CLI can report which files define the same id. The CLI side of this ships with the SDK release that adds `webhook()`. Mono-RevId: 54862ef81ec5adc311aa42783020fedc4b8c3ba9
Explain when to collect a tool result with `addToolOutput` and when to combine `needsApproval` with `execute`. Cover choosing from search results in the human-in-the-loop guide, including server-side selection checks and stable operation IDs for retries. Clarify transcript restoration and the limits of tool-result filtering for external side effects. Update the frontend approval examples and link them to the selection guidance. Mono-RevId: 6c6be1ef71ea4846ba9929179edf48eee77a2af7
Mono-RevId: c893ea0934c69e8fe8ec6f45eaf7be8487e4886b
Clarify chat.agent setup and recovery with examples that check chat ownership before starting sessions or refreshing tokens, use managed streamText for steering, and preserve partial responses after failures. Explain how to load transcripts before resuming the frontend and stop a resumed generation. Update the branching guide to use transcript storage and an explicit active branch, and add checks for transport behavior, error recovery, and branch isolation. Mono-RevId: 071c63091f9605d84806381d97311f2fb1583eea
Mono-RevId: b882af2b96296eb4b1010903fe41af0b2b612572
Improve preview branch auto-archive feature with clearer fields, optional branch exclusions, and smoother modal transitions. Keep branch creation alongside the search controls while simplifying the empty state. Mono-RevId: 8423e6181629877a7971abc9e52cd6fc578a9613
Download saved transcripts from the session inspector when using built-in storage. Downloads preserve the original stored contents, including messages and runtime state, and use a `.jsonl` extension for indexed transcripts. Mono-RevId: 2bd347196eb5993c27d986ebb7abf460020b670e
…imits Concurrency limits can now be paused and resumed from the dashboard, the API, and the SDK (`concurrencyLimits.pause()` / `concurrencyLimits.resume()`), just like queues. A paused limit stops every run holding it from being dequeued while keeping its configured bounds, and resuming restores them. Mono-RevId: 33cbe4dc731671f083779d05cedac03c3709a654
…rencyLimit
## Summary
Marks `queues.overrideConcurrencyLimit` and
`queues.resetConcurrencyLimit` as deprecated in the SDK. These functions
belong to the legacy model where a queue carried its own concurrency
limit. On the current model a queue is only the ordered line runs wait
in, concurrency is declared on the task with the `concurrency` option,
and limits are managed through `concurrencyLimits.override` and
`concurrencyLimits.reset`. The server already rejects these calls for
queues on the current model.
The `@deprecated` JSDoc includes a migration example:
```ts
export const myTask = task({
id: "my-task",
concurrency: { total: 5 },
run: async (payload) => {
// ...
},
});
await concurrencyLimits.override("task/my-task", { total: 10 });
```
Also adds a deprecation callout to the queues docs pointing at the
concurrency docs, plus a changeset.
`queues.pause` and `queues.resume` are intentionally not deprecated:
pausing a queue remains valid flow control on the current model.
Mono-RevId: e75902f159eda3487596ba4d038ed5eedc45bee8
…e dequeue scripts The run queue's self-heal for saturated concurrency sets is now bounded and configurable (page size, lock TTL, passes per dequeue, on/off) and reports its work through OpenTelemetry metrics and span attributes. Defaults preserve existing behaviour apart from capping passes per dequeue script at 2. Mono-RevId: fa05bcd6515f5905f3be52abeebaeeeeb0287617
…ocations atomically Reallocating purchased concurrency across environments now requires billing permissions, matching purchases, and allocations are applied atomically so simultaneous changes can no longer exceed the purchased amount. Mono-RevId: 0ff535b7126ef285de9d815196545022ae2236b5
Add `WEBHOOK_DELIVERIES_REPLICATION_DATABASE_URL` so webhook delivery replication to ClickHouse can use a direct PostgreSQL connection with its own credentials while application queries use a pooler. When unset, replication continues to use `WEBHOOK_DATABASE_URL`, falling back to `DATABASE_URL`. Mono-RevId: 9991ab7c7dd7df8f00e3a93a94ca2b8ad02bf4cf
The active team members list in organization settings is now sorted alphabetically instead of appearing in an arbitrary order. Members are sorted case-insensitively by their display name, falling back to their email address when no name is set. The pending invites list on the same page is now sorted alphabetically by email too. Mono-RevId: 38df496ffef4cf31f8956c4c3cd38387e06e4519
…nd pending version Four docs corrections, one per commit. **Concurrency limits** (`limits.mdx`) — the table published a single headline figure per tier, flattened from the four per-environment values a plan actually sets, and drifted from the current defaults. Now split per environment, with a note that the dashboard Concurrency page is authoritative because the enforced limit also includes purchased add-on and manual concurrency. **Error group alerts** (`troubleshooting-alerts.mdx`) — undocumented until now. They are created from the Errors page, appear in the Alerts table, and are issue-based: they fire on a new issue, a regression or an unignore, and a group that has alerted latches to Unresolved and stays quiet. People configure one expecting a notification per failed run and get silence. Documents the firing rules, the quiet steady state, and which alert type to pick for "tell me about every failed run". Also adds the `alert.error` webhook, which the reference omitted. **Maximum run TTL** (`limits.mdx`) — "all runs have an enforced maximum TTL of 14 days" reads as a cap on how far ahead you can schedule. It isn't: a delayed run arms no TTL expiry, and the TTL clock starts when the delay elapses. Also states that `delay` has no upper bound. **Pending version** (`troubleshooting.mdx`) — a new troubleshooting entry. A run parks here when the version deployed to that environment carries neither the run's task nor its queue. Lists the four causes seen in support: task missing from the deployed version, a queue no deployed task declares, triggering into the wrong org/project/environment, and a later deploy removing or renaming the task or queue while the run was waiting. Also covers runs parked by version skew protection, which wait for their specific external deployment id and expire after 1 hour. Docs only. Mono-RevId: c8a80f8c46ff160ac72fc7c1a1ef5f88a0b02f2c
Mono-RevId: e6399386fe348bde57423668d5427dc7ed9ca935
Mono-RevId: eeea820997ad473c81c42a56fad43b2cc065624e
…rk on zod 3 projects (#4972) ## Summary Deployments built with 4.6.0 to 4.6.3 in projects where `zod` resolves to a 3.x release could not warm start. Every run handed to a warm runner made the runner exit before it started the attempt; the run then waited until the platform's heartbeat redrive requeued it and it started cold, a few minutes late. Cold starts were unaffected, which is why this surfaced as delayed starts rather than errors. ## Root cause The zod v4 migration ([#4039](#4039)) moved core's schemas to `zod/v4`. `snapshotRoute.ts` landed shortly after, still importing the Zod 3 API from `"zod"`, and `SnapshotRouteWire` is composed into `DequeuedMessage` and the worker attempt request bodies. Zod 4 rejects a Zod 3 schema inside a Zod 4 object at parse time, regardless of the input: ``` Invalid element at key "snapshotRoute": expected a Zod schema ``` Inside the monorepo the root `zod` resolves to 4.x, so nothing failed here. In a user's image with zod 3.x installed, the warm-start client's `DequeuedMessage.parse()` threw on every run and the controller exited. ## Fix One import: `snapshotRoute.ts` now uses `zod/v4`. The existing Zod 3 root compatibility test gains a case that bundles `DequeuedMessage` and `WorkerApiRunAttemptStartRequestBody` against a Zod 3 root and parses them; it fails on `main` and passes here. Also verified by packing the built package and parsing a `DequeuedMessage` with `zod@3.25.76` installed.
Mono-RevId: 6d4cc748ca1f7190f58e2983529b65d7326568af
…t webhooks
Directory Sync is now additive: connecting a directory no longer resets
or removes the roles of members you added yourself. A member's role
changes only when they belong to a group you've explicitly mapped to a
role — new groups start unmapped ("Inherit") and change nothing until
you map them. Owner can now be assigned from the Directory Sync and SSO
role menus. Out-of-order webhooks from your identity provider are
retried instead of dropped, so directory setup is more reliable.
Mono-RevId: cd148230d7708141c30a13a4aae0e5ba2a4bfa5e
…ics table API rate limit usage is now recorded per environment in the `metrics` table as `api.rate_limit.allowed`, `api.rate_limit.denied`, `api.rate_limit.remaining_min`, `api.rate_limit.limit.per_second` and `api.rate_limit.limit.burst`, so you can chart requests against your limit and 429s over time on the Query page and in dashboards. Counts are aggregated in the rate-limit middleware and written as ClickHouse async inserts, so recording adds no per-request I/O. Off by default; enable with `API_RATE_LIMIT_METRICS_ENABLED=1`, or `allowlist` to record only organizations opted in through the `apiRateLimitMetricsEnabled` feature flag. Mono-RevId: 5cc33e4c426f673c694938cfd4eda686b591a7d0
…y dequeue (#4367) Off by default. When many concurrency-key variants share one task queue, the dequeue serves the oldest waiting run first, so one key's large backlog is served to exhaustion while keys queued behind it wait for the whole pile to drain. This adds an opt-in fair order: each key gets a virtual clock, the dequeue serves the smallest clock and advances it, so keys take turns instead of one pile draining. With the flag off, the existing scripts run unchanged.  ## How it works  - Three keys per base queue: `:ckVtime` holds a virtual-time tag per variant, `:ckVtimeFloor` is the current virtual time, and `:ckVtimeIdle` remembers the tags of variants that have drained. `ckIndex` keeps its head-timestamp domain, so time-eligibility, master-queue rebalancing and every other writer stay untouched, which is what makes it mixed-deploy safe. - A flag-selected two-pass dequeue. Pass 1 serves the lowest tags and charges each serve a quantum, bounded by a window of `maxCount * RUN_ENGINE_CK_VTIME_WINDOW_MULTIPLIER`. Pass 2 fills any leftover slots in today's age order and discovers variants that have no tag yet, so the command is a strict superset of today and stays work-conserving. - Only pass-1 serves move the floor, so a variant that cannot be served cannot drag the clock along behind it. - A draining variant parks its tag in the idle set and takes it back on its next enqueue or nack, so draining and returning isn't a way to reset your clock. All four routes out park it: the dequeue, ack, dead-letter and TTL expiry. - A brand-new variant joins one quantum behind the current leader rather than at the floor. Registering at the floor is what let a tenant sharding across fresh keys outrank everything already waiting, and it's now reserved for repair (pass-2 discovery and the gated batch), where everything in sight is established work that lost its tag. - A candidate that can't be served this call, because its head is scheduled in the future or it's at its per-key concurrency ceiling, says so instead of spending one of pass 1's window slots. - New behaviour lives only in new Lua command names. The existing enqueue/dequeue/nack scripts are byte-for-byte unchanged, which is checkable by hashing each script body, so flag-off is identical to today. ## Numbers Fairness, flag OFF against ON under the same load on the same box, wait measured in logical dequeue steps: | scenario | victim wait p99 | Jain fairness | | --- | --- | --- | | skewed backlog | 636 -> 196 (-69%) | 0.34 -> 1 | | trickle behind a backlog | 596 -> 176 (-70%) | 0.50 -> 1 | A fresh key minted per run against a 2000-deep backlog: the backlogged variant went from 1 slot in 600 to 120 in 600, which is what age order would have given it. Cost, measured saturated with the generator next to Redis, 1M invocations an arm: | path | flag-on cost | scaling | | --- | --- | --- | | enqueue | +1.8 usec | flat, 100 to 50k keys | | dequeue, serving | +3 to 4% call p95 | from the fairness runs above | | dequeue, every variant gated | 33.5 usec at 1k keys | 36.4 at 10k | | memory at rest | none | the idle set doesn't exist until something drains | | memory under churn | ~1.5MB per queue | capped by rank at 10k entries | Rule of thumb that fell out of the sweep, useful for costing anything else added to these scripts: about 1.38 usec fixed per EVALSHA plus 0.33 per `redis.call`. ## Testing 176 tests across the run-queue suite, green with the flag off and on. Fairness is proven on the real batched dequeue path rather than one message per call, plus multi-consumer exactly-once, a per-dequeue op-count budget, and behaviour tests for the floor, tag advance, GC, registration and the credit round trip through every drain path. Two mutation audits (33 mutations, one change to the production Lua at a time, rerun the suites, on the principle that a green run against a broken invariant is a hole). They found six things nothing was checking, all now covered: the pass-1 guard for a future-scheduled head, the idle park on ack, dead-letter and TTL expiry, nack's credit restore, the dead-letter caller, the TTL enqueue's own copy of the registration block, and any quantum other than 1. ## Rollout Off by default behind `RUN_ENGINE_CK_VTIME_SCHEDULING_ENABLED`, with `RUN_ENGINE_CK_VTIME_QUANTUM`, `RUN_ENGINE_CK_VTIME_WINDOW_MULTIPLIER` and `RUN_ENGINE_CK_VTIME_STATE_TTL_SECONDS` for the knobs. Enable on a staging cell, then production. Rollback is flipping the flag off, and leftover state expires within a day. During a rolling deploy, old instances serve in age order and are folded in by pass 2, so nothing is lost and no run is served twice. Every mutation is a single atomic Lua script, which is what makes those interleavings safe. That atomicity assumes the single-node Redis the run queue actually runs on: it has no cluster-mode setting (every other Redis in `env.server.ts` has one, `RUN_ENGINE_RUN_QUEUE_REDIS_*` doesn't), and the master queue key sits outside the base queue's hash slot exactly as it does in the command this one is modelled on. ## Known limitations **Registration order sets relative priority.** Within the fair pass a variant's tag comes from when it joined and how much it has been served, so message age doesn't break in. A key that arrives during a busy period sits a quantum behind the leader and keeps that place until the floor catches up. This is deliberate, since the alternative (strict fair share) makes minting fresh keys pay, and it's worth confirming against a real high-cardinality workload before the flag goes on anywhere. **Ties break lexically.** Variants on the same tag, at a cold start or when a collected variant re-registers, are served in queue-name order. It's a pre-existing effect of the old head-timestamp ordering and only affects who goes first, not long-run fairness. **The vtime scripts are copies.** Seven pairs, about 688 identical Lua lines, kept as copies so the flag-off path stays byte-identical. A change to one of the originals doesn't reach its copy, which already happened once on this branch (main added a TTL re-registration to `dequeueMessagesFromCkQueueTracked` and the copy went without it until 7f01aaf). A follow-up PR will pin each copy against its original so drift shows up as a red check.
Stream closure alone cannot confirm that a specific action's input was processed; this adds an optional per-call `onSettled` callback to `TriggerChatTransport.sendAction()`. ## ✅ Checklist - [x] I have followed every step in the [contributing guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md) - [x] The PR title follows the convention - [x] I ran and tested the code works --- ## Testing SDK build, 770 SDK tests (including 45 real-HTTP settlement tests), ESM/CommonJS package smoke checks, formatting, lint, and Knip passed; live-worker E2E was not run. --- ## Changelog Adds validated, once-per-call settlement with cancellation and supersession safeguards, public types, API documentation, and a patch changeset. Documents that settlement does not prove application success and that a missing callback does not make retrying safe. --- ## Screenshots Not applicable — no visual changes 💯
Improve the first GitHub deployment experience: Deploy now explains when a branch doesn't exist on GitHub, a harmless first-build cache message no longer shows as an error, the deployment panel stays on screen after the first deploy finishes, the empty development Tasks page uses the new setup layout, and the deployment setup screen is vertically centered. Mono-RevId: 07d4623e6fbe912906e1976f513f962c5ec42aa6
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
17 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #
✅ Checklist
Testing
[Describe the steps you took to test this change]
Changelog
[Short description of what has changed]
Screenshots
[Screenshots]
💯