Skip to content

ci: use persistent Incus workers for trusted heavy builds - #2090

Draft
NathanFlurry wants to merge 5 commits into
mainfrom
codex/incus-heavy-ci
Draft

NathanFlurry wants to merge 5 commits into
mainfrom
codex/incus-heavy-ci

Conversation

@NathanFlurry

@NathanFlurry NathanFlurry commented Nov 4, 2026 •

Copy link
Copy Markdown
Member

Route trusted main-branch heavy Rust, WASM and native Docker jobs to persistent Incus runners while retaining hosted runners for pull requests, non-main previews, small jobs and native Linux ARM.

  • Keep compiler/toolchain caches outside checkout, isolate patched WASM vendor/sysroot state by tracked inputs, and namespace mutable Docker caches by runner slot, platform and profile.
  • Preserve the sysroot cache symlink during clean rebuilds, and honor the worker compiler-job budget in the C/LLVM toolchain scripts.
  • Use the persistent VM-local BuildKit daemon with job-specific client settings and extraction cleanup. Preserve the narrow organization runner-group workflow allowlist.
  • Validate all accepted ci-02 and ci-03 slots with a build-only smoke workflow. Optional manual checks exercise four simultaneous real Linux builds, Darwin cross compilation, and patched Rust/C WASM commands; they do not publish artifacts or releases.

Validation: all seven registered slots passed Cargo/WASI binary reuse, offline pnpm reuse, BuildKit layers/mutable caches and actual active-job GC deferral. Four simultaneous Linux builds passed matching warm executables, dependency reuse, glibc compatibility and container smoke without cgroup OOM events. Both Darwin architectures passed real builds and matching warm executables with external dependency reuse in cross-build validation. YAML, shell and embedded Python syntax checks pass.

The remaining dedicated workers are still being provisioned. Patched Rust/C WASM validation passed warm binary equality, Cargo Fresh, C curl reuse and all cache-link persistence checks. A second job attempt also passed after a fresh checkout. The complete rollout remains in progress; this PR stays a draft. The temporary branch-specific smoke-workflow allowlist entry must be removed after rollout or replaced with the main reference after merge. No publishing workflow was dispatched.

@railway-app

railway-app Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

🚅 Environment agentos-pr-2090 in rivet-frontend has no services deployed.

3 services not affected by this PR
  • kitchen-sink
  • mcp-hub
  • sentry-tunnel

@github-actions

github-actions Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

📟 Pip-Boy · 6fb2782

📦 Packages · fresh npm install resolve, no lockfile · this PR vs base

No notable install size, dependency, or entrypoint changes across 6 packages.

All 6 packages · linux-x64-gnu
package (linux-x64-gnu) @latest base this PR Δ vs base packages
@rivet-dev/agentos-core 326.2 MB 326.3 MB 326.3 MB 0 10 (0)
@rivet-dev/agentos-node-pty 7.6 KB 18.0 KB 18.0 KB 0 1 (0)
@rivet-dev/agentos-posix 254.0 MB 254.0 MB 254.0 MB 0 7 (0)
@rivet-dev/agentos-runtime-core 254.0 MB 254.0 MB 254.0 MB 0 6 (0)
@rivet-dev/agentos-sandbox 363.9 MB 363.9 MB 363.9 MB 0 85 (0)
@rivet-dev/agentos-toolchain 130.8 KB 141.2 KB 141.2 KB 0 2 (0)

🧹 Rust dependencies · cargo deny + cargo shear

check new in this PR fixed existing on base
advisories ✅ 0 0 52
licenses ✅ 0 0 0
sources ✅ 0 0 0
bans ✅ 0 0 0
unused ✅ 0 0 17

🦀 Binaries · publish Dockerfiles, --release

Base: main@e57c6aa, the newest main commit with native builds before the merge base. Size changes merged to main since then show up here too.

artifact npm @latest base this PR Δ vs base
agentos-native-sidecar darwin-arm64 120.4 MB 120.4 MB 120.4 MB 0
agentos-native-sidecar darwin-x64 129.7 MB 129.7 MB 129.7 MB 0
agentos-native-sidecar linux-arm64-gnu 141.2 MB 141.2 MB 141.2 MB 0
agentos-native-sidecar linux-x64-gnu 144.0 MB 144.0 MB 144.0 MB 0

⏱ Timing

job wall time slowest step
zizmor 9s checkout · 2s
changes 24s checkout · 20s
self-test 32s Run pnpm --filter pipboy test · 10s
hygiene 55s checkout · 22s
footprint 1m28s Pack this commit · 28s
native · linux-x64-gnu 11m57s Build with the publish Dockerfile · 11m36s
native · darwin-x64 9m27s Build with the publish Dockerfile · 9m00s
native · darwin-arm64 9m21s Build with the publish Dockerfile · 8m49s
native · linux-arm64-gnu 9m41s Build with the publish Dockerfile · 9m13s
report 22s Write the comment and run summary · 9s
🔭 Full CI trace for this commit

One OpenTelemetry trace of CI and Pip-Boy for 6fb2782: every job and step, with the sizes above attached. Run after CI finishes.

pnpm install --frozen-lockfile --filter 'pipboy...'
pnpm --filter pipboy pipboy trace --sha 6fb2782d9a948ab903667c90e87e6b5b51d46665

Add --otlp <url> to send it to any OTLP/HTTP collector.

Agent instruction: From a rivet-dev/agentos checkout, run the commands above, then read the trace file the last command prints. List the slowest jobs and steps and explain where CI time went.

@NathanFlurry
NathanFlurry force-pushed the codex/incus-heavy-ci branch 10 times, most recently from 84dd6a6 to 6c0a801 Compare November 18, 2026 06:45

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant