diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 7fcb0c20b..3f5505697 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -159,9 +159,19 @@ jobs: # [env] block (cargo has no [target..env] — the macOS section in that file # is inert and cargo warns "unused key"), so before that change this job pointed # bindgen at the win64 tree and could never have gone green. + # + # Without a library on ORT_DYLIB_PATH, `runtime_available()` is false and every + # segmentation test returns early — the suite goes green having exercised no + # inference at all, which is exactly how the `ort`-panics-when-absent bug got + # in. Staging it here is what makes `the_whole_loop_produces_a_mask_from_ + # compose_frame_alone` a real test on this runner instead of a skipped one. + # ~30 MB, next to nothing beside `brew install ffmpeg` above. + - name: Stage ONNX Runtime + run: node scripts/fetch-onnxruntime.mjs - name: cargo test (compositor, aarch64-apple-darwin) env: MAC_FFMPEG_DIR: /opt/homebrew/opt/ffmpeg + ORT_DYLIB_PATH: ${{ github.workspace }}/electron/native/bin/darwin-arm64/libonnxruntime.dylib run: | cd crates cargo test -p openscreen-compositor --lib --tests diff --git a/THIRD-PARTY-NOTICES.md b/THIRD-PARTY-NOTICES.md index 330bf7492..f10c009c6 100644 --- a/THIRD-PARTY-NOTICES.md +++ b/THIRD-PARTY-NOTICES.md @@ -49,6 +49,42 @@ distributed by their own registries, not redistributed inside our binaries. - The speech model (`ggml-*.bin`) is **not** bundled — it is downloaded into the user's data directory on first use by `electron/stt/modelManager.ts`. +## ONNX Runtime (Windows and Apple Silicon macOS) + +- **Component**: `onnxruntime.dll` / `libonnxruntime.dylib`, under + `resources/electron/native/bin/-/`. +- **License**: MIT — . +- Not built here: the pinned upstream release archive is downloaded, SHA-256 + verified and unpacked by `scripts/fetch-onnxruntime.mjs`, which also checks the + archive's own LICENSE really is MIT before vendoring anything. +- **Why it ships**: the native compositor segments the webcam subject with it, on + the CPU execution provider, to drive the camera background cutout/blur/custom + modes. The `gpu_cuda*` builds are deliberately not used — they are an order of + magnitude larger and carry NVIDIA redistribution terms. +- **Not on Intel macOS**: upstream publishes no `osx-x86_64` asset from 1.27 on, + so the x64 DMG ships without it and the camera background effects are simply + absent there. Not shipped on Linux either, where the compositor has no capture + path for the mask yet. +- The segmentation model it runs is a separate component, immediately below. + +## MediaPipe Selfie Segmentation — model weights + +- **Components**: `selfie_segmentation.tflite`, + `selfie_segmentation_landscape.tflite` and the `selfie_segmentation_landscape.onnx` + derived from them, shipped inside `app.asar` under `dist/mediapipe/`. +- **License**: Apache-2.0 — . + Copyright The MediaPipe Authors. +- The `.onnx` is a **derived work**, generated from the vendored `.tflite` by + `scripts/convert-selfie-segmentation-to-onnx.py`. No third-party weights are + downloaded at build time. +- **Why it is listed here**: these weights are redistributed inside the installer, + and Apache-2.0 §4 asks that the attribution travel with them. The provenance note + in `public/mediapipe/selfie_segmentation/README.md` does not — electron-builder's + `"!*.md"` filter strips it from the package — so this file is the only copy a user + ever receives. +- The MediaPipe **JavaScript** solution and its two ~5.6 MB WASM builds are no longer + bundled: inference moved into the native compositor, and nothing loaded them. + ## Microsoft OpenMP runtime — `vcomp140.dll` (Windows only) - **Component**: `resources/electron/native/bin/win32-x64/vcomp140.dll`. diff --git a/package.json b/package.json index 056e4aecb..f48fefe5a 100644 --- a/package.json +++ b/package.json @@ -42,15 +42,15 @@ "assets:appx": "node scripts/generate-appx-assets.mjs", "preview": "vite preview", "build:native:mac": "node scripts/build-macos-screencapturekit-helper.mjs", - "build:mac": "npm run build:native:mac && npm run fetch:ffmpeg:mac && npm run build:native:compositor:mac && tsc && vite build && electron-builder --mac", + "build:mac": "npm run build:native:mac && npm run fetch:ffmpeg:mac && npm run fetch:onnxruntime && npm run build:native:compositor:mac && tsc && vite build && electron-builder --mac", "build:native:win": "node scripts/build-windows-wgc-helper.mjs", "stage:vcomp": "node scripts/stage-vcomp-runtime.mjs", "build:native:compositor": "node scripts/build-windows-compositor-addon.mjs", "build:native:compositor:mac": "node scripts/build-macos-compositor-addon.mjs", "build:native:compositor:linux": "node scripts/build-linux-compositor-addon.mjs", "build:native:linux": "node scripts/build-linux-pipewire-helper.mjs", - "build:win": "npm run build:native:win && npm run fetch:ffmpeg && npm run stage:vcomp && npm run build:native:compositor && tsc && vite build && electron-builder --win --config.npmRebuild=false", - "build:win:store": "npm run build:native:win && npm run fetch:ffmpeg && npm run stage:vcomp && npm run build:native:compositor && tsc && vite build && electron-builder --win appx --config.npmRebuild=false", + "build:win": "npm run build:native:win && npm run fetch:ffmpeg && npm run fetch:onnxruntime && npm run stage:vcomp && npm run build:native:compositor && tsc && vite build && electron-builder --win --config.npmRebuild=false", + "build:win:store": "npm run build:native:win && npm run fetch:ffmpeg && npm run fetch:onnxruntime && npm run stage:vcomp && npm run build:native:compositor && tsc && vite build && electron-builder --win appx --config.npmRebuild=false", "build:linux": "npm run fetch:ffmpeg:sdk && npm run build:native:linux && npm run build:native:compositor:linux && tsc && vite build && electron-builder --linux AppImage deb pacman rpm --config.npmRebuild=false", "build:whisper-binaries": "bash scripts/build-whisper-stt.sh", "test:whisper-stt": "node scripts/test-whisper-stt.mjs", @@ -92,7 +92,8 @@ "prepare": "husky", "fetch:ffmpeg": "node scripts/fetch-ffmpeg.mjs", "fetch:ffmpeg:mac": "node scripts/fetch-ffmpeg-macos.mjs", - "fetch:ffmpeg:sdk": "node scripts/fetch-ffmpeg.mjs --sdk-only" + "fetch:ffmpeg:sdk": "node scripts/fetch-ffmpeg.mjs --sdk-only", + "fetch:onnxruntime": "node scripts/fetch-onnxruntime.mjs" }, "dependencies": { "@fix-webm-duration/fix": "^1.0.1", diff --git a/scripts/fetch-onnxruntime.mjs b/scripts/fetch-onnxruntime.mjs new file mode 100644 index 000000000..c295f750e --- /dev/null +++ b/scripts/fetch-onnxruntime.mjs @@ -0,0 +1,307 @@ +// Provisions the ONNX Runtime shared library into +// electron/native/bin/-/, next to the compositor addon and the +// ffmpeg libraries. That directory is gitignored and shipped by electron-builder's +// extraResources, so this runs at build time rather than committing a 15-38 MB binary. +// +// WHY IT SHIPS: the native compositor segments the webcam subject on the CPU +// execution provider (crates/compositor/src/segmentation.rs), and `ort` is linked +// with `load-dynamic` — nothing is needed to BUILD, but at runtime +// `ensureOnnxRuntimeOnPath` (electron/native-bridge/services/compositorViewService.ts) +// walks this exact directory looking for the library and sets ORT_DYLIB_PATH to it. +// Without it `Segmenter::load` fails, the compositor logs one line and draws the +// webcam unsegmented — so the AI background cutout/blur/custom modes are simply off. +// Everything degrades; nothing breaks. That is why this script never fails a build. +// +// VERSION IS NOT FREE TO MOVE. crates/Cargo.toml pins `ort` with feature `api-NN`, +// which is the MINIMUM ONNX Runtime minor version the crate will accept — a lower +// one makes `GetApi` return null and `ort` panics rather than erroring. The pin here +// must satisfy that, and scripts/fetch-onnxruntime.test.mjs cross-checks the two so +// a bump on either side cannot land alone. +// +// SUPPLY CHAIN. This binary is signed and shipped to every user, so nothing floats: +// - Pinned to an immutable release tag, never `latest`. +// - SHA-256 verified before the archive is opened. The digests below are the ones +// GitHub publishes per asset (`digest` in the releases API), independently +// re-verified by downloading and hashing. +// - Only the plain CPU assets. The `gpu_cuda*` variants are 200-320 MB and pull +// NVIDIA runtime dependencies we neither need nor may redistribute; the measured +// decision to use the CPU EP is in +// technical-documentation/engineering/webcam-segmentation.md. +// +// LICENSING: ONNX Runtime is MIT, which is compatible with this MIT app — but the +// archive is checked rather than trusted, the same way fetch-ffmpeg.mjs verifies +// ffmpeg's LGPL-ness instead of believing the asset name. Attribution ships in +// THIRD-PARTY-NOTICES.md. + +import { spawnSync } from "node:child_process"; +import crypto from "node:crypto"; +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { fileURLToPath } from "node:url"; + +const __dirname = path.dirname(fileURLToPath(import.meta.url)); +const ROOT = path.join(__dirname, ".."); + +/** + * The pinned release. Asset names are DERIVED from it rather than written out per + * entry, which is deliberate: fetch-ffmpeg.mjs keeps full asset strings and grew a + * test because a re-pin moved some and not others. Templating removes that failure + * mode by construction. The digests still have to move by hand — but a stale one + * fails loudly on the SHA-256 check before anything is extracted, which is the safe + * direction to fail in. + */ +const VERSION = "1.27.1"; +const BASE = `https://github.com/microsoft/onnxruntime/releases/download/v${VERSION}`; + +/** + * Per-target: the upstream artifact slug, its digest, and the library to lift out. + * + * `out` is not cosmetic — it is the exact name `ortLibName()` looks for in + * compositorViewService.ts. `member` is the file inside the archive, which on macOS + * and Linux is the VERSIONED real file rather than the unversioned symlink beside + * it: tar restores that symlink as a symlink, and a dangling one in the packaged app + * would resolve to nothing. + * + * darwin-x64 is absent and cannot be added: Microsoft publishes no `osx-x86_64` + * (or universal) asset for any release from 1.27 on — arm64 is the only macOS + * target. Building it from source is the only way to change that, and it is not + * worth an ffmpeg-macos-sized build script for a shrinking platform when the + * fallback is "the effect is off". See the darwin-x64 branch in main(). + */ +const PINNED = { + "win32-x64": { + slug: "win-x64", + ext: "zip", + sha256: "2e00414a63fdef0914cd5a5ede6c707844878e0c08e1b6693842f0451b2df2a1", + member: "onnxruntime.dll", + out: "onnxruntime.dll", + }, + "win32-arm64": { + slug: "win-arm64", + ext: "zip", + sha256: "6e22c2061ba6400b42a59663d700c8694e4e8fe654cf452c4700c24237407ae1", + member: "onnxruntime.dll", + out: "onnxruntime.dll", + }, + "darwin-arm64": { + slug: "osx-arm64", + ext: "tgz", + sha256: "e42b77a7281cc6e55141bf44fcfbac2c782b823a491bbb6ac33c781dd991f8a6", + member: `libonnxruntime.${VERSION}.dylib`, + out: "libonnxruntime.dylib", + }, + // Linux is pinned but not yet wired into `build:linux`: the Linux compositor + // back-end carries the segmentation SHADER only — nothing captures the webcam + // frame or uploads a mask there, so `fx.z` never leaves 0 and the library would + // be 23 MB of installer for a code path that cannot run. The entry is here so + // that landing the Linux capture half is a one-line change to package.json. + // See technical-documentation/engineering/webcam-segmentation-backend-port.md. + "linux-x64": { + slug: "linux-x64", + ext: "tgz", + sha256: "25b1ef1fea1acd210d63f8f24dc870ad6e077795ce1f54876252c6d3803c15af", + member: `libonnxruntime.so.${VERSION}`, + out: "libonnxruntime.so", + }, + "linux-arm64": { + slug: "linux-aarch64", + ext: "tgz", + sha256: "33c67e33d1e25b816878366ea276589a024f71f000e7ff955c4b33224d639edd", + member: `libonnxruntime.so.${VERSION}`, + out: "libonnxruntime.so", + }, +}; + +const assetName = (spec) => `onnxruntime-${spec.slug}-${VERSION}.${spec.ext}`; + +/** Magic bytes the vendored library must start with, per target platform. */ +const MAGIC = { + win32: { bytes: [0x4d, 0x5a], name: "PE (MZ)" }, // .dll + darwin: { bytes: [0xcf, 0xfa, 0xed, 0xfe], name: "Mach-O 64" }, // .dylib + linux: { bytes: [0x7f, 0x45, 0x4c, 0x46], name: "ELF" }, // .so +}; + +function run(cmd, args, opts = {}) { + return spawnSync(cmd, args, { stdio: "inherit", ...opts }); +} + +function tarBin() { + if (process.platform !== "win32") return "tar"; + const sys32 = path.join(process.env.SystemRoot ?? "C:\\Windows", "System32", "tar.exe"); + return fs.existsSync(sys32) ? sys32 : "tar"; +} + +function extract(archive, destDir) { + fs.mkdirSync(destDir, { recursive: true }); + // Run from destDir with a bare filename: given an absolute Windows path, tar + // reads "C:\..." as host:path and tries to resolve a host called C. + const r = run(tarBin(), [archive.endsWith(".zip") ? "-xf" : "-xzf", path.basename(archive)], { + cwd: destDir, + }); + if (r.status !== 0) throw new Error(`tar failed to extract ${path.basename(archive)}`); +} + +/** Depth-first search for a file by exact basename. */ +function find(dir, name) { + for (const entry of fs.readdirSync(dir, { withFileTypes: true })) { + const p = path.join(dir, entry.name); + if (entry.isDirectory()) { + const hit = find(p, name); + if (hit) return hit; + } else if (entry.name === name) { + return p; + } + } + return null; +} + +/** + * Refuses anything that is not the MIT ONNX Runtime we pinned. + * + * Three independent checks, because the digest alone only proves we got the archive + * we asked for — it says nothing about having lifted the RIGHT FILE out of it, which + * is where a re-pin actually goes wrong (a renamed member silently vendors a 20 KB + * provider stub, and the failure surfaces as "the effect does nothing" months later). + * + * 1. the archive's LICENSE really is MIT — asset names are not evidence; + * 2. the library is a binary of the expected format for the target platform; + * 3. it carries the pinned version string, which is what `GetVersionString()` + * returns and what `ort` compares against its `api-NN` floor. + */ +function verify(libPath, licensePath, targetPlatform) { + const license = fs.readFileSync(licensePath, "utf8"); + if (!/^MIT License/m.test(license)) { + throw new Error( + `${path.basename(licensePath)} does not begin with "MIT License".\n` + + "Refusing to vendor: ONNX Runtime is MIT and this app is MIT — a relicensed\n" + + "upstream is a decision for a human, not a build script.", + ); + } + + const buf = fs.readFileSync(libPath); + const magic = MAGIC[targetPlatform]; + if (!magic.bytes.every((b, i) => buf[i] === b)) { + const got = [...buf.subarray(0, 4)].map((b) => b.toString(16).padStart(2, "0")).join(" "); + throw new Error( + `${path.basename(libPath)} is not a ${magic.name} binary (starts with ${got}).\n` + + "The archive layout probably changed under the pin — check `member`.", + ); + } + + // The version lives in the binary as a plain NUL-terminated string. + if (!buf.includes(Buffer.from(`\0${VERSION}\0`, "latin1"))) { + throw new Error( + `${path.basename(libPath)} does not carry the version string ${VERSION}.\n` + + "Either the pin and the digest disagree, or the wrong member was extracted.", + ); + } + + return `MIT ONNX Runtime ${VERSION}, ${magic.name}, ${(buf.length / 1048576).toFixed(1)} MB`; +} + +async function download(spec) { + const asset = assetName(spec); + const tmp = fs.mkdtempSync(path.join(os.tmpdir(), "openscreen-ort-")); + console.log(`Downloading ${asset}\n from v${VERSION}`); + const res = await fetch(`${BASE}/${asset}`); + if (!res.ok) throw new Error(`Download failed: ${res.status} ${res.statusText}`); + const bytes = Buffer.from(await res.arrayBuffer()); + + // Before opening it: is this the exact artifact we pinned? + const got = crypto.createHash("sha256").update(bytes).digest("hex"); + if (got !== spec.sha256) { + fs.rmSync(tmp, { recursive: true, force: true }); + throw new Error( + `SHA-256 mismatch for ${asset}\n expected ${spec.sha256}\n got ${got}\n` + + "Refusing to extract. Either the pin is stale or the artifact changed under it.", + ); + } + console.log(` sha256 ok (${(bytes.length / 1048576).toFixed(0)} MB)`); + + const archive = path.join(tmp, asset); + fs.writeFileSync(archive, bytes); + extract(archive, tmp); + return tmp; +} + +async function main() { + // `--target` exists for CI, which provisions for the runner it is on; without it + // the host is the target, which is what every local build wants. + const targetArg = process.argv.find((a) => a.startsWith("--target=")); + const tag = targetArg + ? targetArg.slice("--target=".length) + : `${process.platform}-${process.arch}`; + const [targetPlatform] = tag.split("-"); + + // Not an error, and deliberately exit 0: `build:mac` runs on an Intel runner for + // the x64 DMG, and there is no upstream library to give it. Failing here would + // break a release build over a feature that is designed to be absent gracefully. + if (tag === "darwin-x64") { + console.log( + "ONNX Runtime is not provisioned for darwin-x64: Microsoft publishes no\n" + + "osx-x86_64 (or universal) asset for 1.27 or later — arm64 is the only macOS\n" + + "target. The webcam background effects are therefore OFF on Intel Macs; the\n" + + "compositor logs one line and draws the camera unsegmented. Nothing else changes.", + ); + return; + } + + const spec = PINNED[tag]; + if (!spec) { + console.log( + `No pinned ONNX Runtime for ${tag} — skipping. Have: ${Object.keys(PINNED).join(", ")}`, + ); + return; + } + if (!MAGIC[targetPlatform]) { + throw new Error(`Unknown target platform in --target=${tag}`); + } + + const binDir = path.join(ROOT, "electron", "native", "bin", tag); + const dest = path.join(binDir, spec.out); + + if (fs.existsSync(dest) && !process.argv.includes("--force")) { + // Re-verify rather than trusting the filename: this directory is gitignored + // scratch space that a half-finished run or a hand copy can leave anything in. + const buf = fs.readFileSync(dest); + const magic = MAGIC[targetPlatform]; + const looksRight = + magic.bytes.every((b, i) => buf[i] === b) && + buf.includes(Buffer.from(`\0${VERSION}\0`, "latin1")); + if (looksRight) { + console.log(`Already present: ${dest}`); + console.log(` ONNX Runtime ${VERSION}, ${(buf.length / 1048576).toFixed(1)} MB`); + console.log("Use --force to re-download."); + return; + } + console.log(`Present but not ONNX Runtime ${VERSION} — re-fetching: ${dest}`); + } + + const tmp = await download(spec); + try { + const lib = find(tmp, spec.member); + if (!lib) throw new Error(`${spec.member} not found inside ${assetName(spec)}`); + const license = find(tmp, "LICENSE"); + if (!license) throw new Error(`LICENSE not found inside ${assetName(spec)}`); + + // Verify BEFORE vendoring: nothing unchecked reaches electron/native/bin, + // where the packager would happily ship it. + console.log("Verifying..."); + const banner = verify(lib, license, targetPlatform); + + fs.mkdirSync(binDir, { recursive: true }); + fs.copyFileSync(lib, dest); + if (targetPlatform !== "win32") fs.chmodSync(dest, 0o755); + + console.log(` ${banner}`); + console.log(`\nVendored -> ${dest}`); + } finally { + fs.rmSync(tmp, { recursive: true, force: true }); + } +} + +main().catch((err) => { + console.error(`\n${err.message}`); + process.exit(1); +}); diff --git a/scripts/fetch-onnxruntime.test.mjs b/scripts/fetch-onnxruntime.test.mjs new file mode 100644 index 000000000..6424e4c7f --- /dev/null +++ b/scripts/fetch-onnxruntime.test.mjs @@ -0,0 +1,116 @@ +// The ONNX Runtime pin is coupled to a pin in a DIFFERENT LANGUAGE, and nothing +// else notices when they drift apart. +// +// `crates/Cargo.toml` gives `ort` the feature `api-NN`. That NN is the minimum ONNX +// Runtime minor version the crate accepts: `ort_sys` computes `ORT_API_VERSION` from +// it and asks the library for that API, and a library older than NN returns null — +// at which point `ort` PANICS rather than erroring (it is documented doing so in +// segmentation.rs, and it took down a render thread once already). So bumping `ort` +// without re-pinning this script ships a build where the effect is not merely off +// but actively fatal on the first frame that asks for it. +// +// The reverse drift is quieter and worse: pinning a NEWER runtime than the crate was +// built for makes `ort` log a compatibility warning to stderr and carry on, which in +// a packaged Electron app nobody reads. +// +// Neither direction is visible in review — the two lines are in different files, in +// different languages, edited by different tasks. This test is the thing that sees it. +// +// Read as source text rather than imported: fetch-onnxruntime.mjs calls main() at +// import and would start downloading. The property under test is a property of the +// literal table anyway. + +import fs from "node:fs"; +import path from "node:path"; +import { fileURLToPath } from "node:url"; +import { describe, expect, it } from "vitest"; + +const HERE = path.dirname(fileURLToPath(import.meta.url)); +const source = fs.readFileSync(path.join(HERE, "fetch-onnxruntime.mjs"), "utf8"); +const cargoToml = fs.readFileSync(path.join(HERE, "..", "crates", "Cargo.toml"), "utf8"); + +const version = source.match(/^const VERSION = "([^"]+)";/m)?.[1]; +// Anchored at the property so the file's (extensive) prose cannot match. +const digests = [...source.matchAll(/^\s*sha256:\s*"([^"]+)"/gm)].map((m) => m[1]); +const slugs = [...source.matchAll(/^\s*slug:\s*"([^"]+)"/gm)].map((m) => m[1]); +const tags = [...source.matchAll(/^\t"([a-z0-9]+-[a-z0-9]+)":\s*\{$/gm)].map((m) => m[1]); + +/** The `api-NN` feature `crates/Cargo.toml` gives `ort`, as a number. */ +const ortApiFloor = () => { + const block = cargoToml.match(/^ort = \{[\s\S]*?^\]\s*\}/m)?.[0] ?? ""; + const found = [...block.matchAll(/"api-(\d+)"/g)].map((m) => Number(m[1])); + return found.length ? Math.max(...found) : null; +}; + +describe("fetch-onnxruntime pins", () => { + // Without this, a reformat that breaks the regexes above would leave every other + // assertion iterating an empty array and passing vacuously. + it("still finds the pin table", () => { + expect(version, "VERSION not found in fetch-onnxruntime.mjs").toMatch(/^\d+\.\d+\.\d+$/); + expect(slugs.length).toBeGreaterThanOrEqual(4); + expect(digests).toHaveLength(slugs.length); + expect(tags).toHaveLength(slugs.length); + }); + + // THE point of this file. + it("pins a runtime that satisfies the `api-NN` floor in crates/Cargo.toml", () => { + const floor = ortApiFloor(); + expect(floor, "no api-NN feature found on `ort` in crates/Cargo.toml").toBeGreaterThan(0); + const minor = Number(version.split(".")[1]); + expect( + minor, + `crates/Cargo.toml asks ort for api-${floor}, so ONNX Runtime must be >= 1.${floor}.x, ` + + `but fetch-onnxruntime.mjs pins ${version}. A runtime below the floor makes GetApi ` + + "return null and ort PANICS. Re-pin VERSION and every sha256 together.", + ).toBeGreaterThanOrEqual(floor); + }); + + // Above the floor is not free either: ort warns at load and carries on, which in a + // packaged app goes to a stderr nobody reads. Exact match is the intended state. + it("pins the runtime the crate was actually built for, not merely a compatible one", () => { + const floor = ortApiFloor(); + expect( + Number(version.split(".")[1]), + `ONNX Runtime ${version} does not match the api-${floor} ort was built against. ` + + "Below it, ort panics; above it, ort logs a compatibility warning at every " + + "startup, into a stderr no packaged app shows. If the mismatch is deliberate, " + + "move ort to the matching api-NN feature in the same change.", + ).toBe(floor); + }); + + // The gpu_cuda variants are 200-320 MB and carry NVIDIA runtime redistribution + // terms. Nothing in this app uses a GPU execution provider — the CPU EP was the + // measured choice (webcam-segmentation.md), and it is what makes ONNX Runtime + // shippable at all. + it("pins only the plain CPU assets", () => { + for (const slug of slugs) { + expect(slug, `${slug} is not a plain CPU asset`).not.toMatch( + /gpu|cuda|tensorrt|qnn|training/, + ); + } + }); + + it("pins a full sha-256 for every asset", () => { + for (const digest of digests) { + expect(digest).toMatch(/^[0-9a-f]{64}$/); + } + }); + + // The keys are matched against `${process.platform}-${process.arch}`, so a + // plausible-looking typo (`darwin-aarch64`, `win-x64`) silently provisions + // nothing and the effect is off with no error anywhere. + it("keys the table by real process.platform-process.arch tags", () => { + for (const tag of tags) { + expect(tag).toMatch(/^(win32|darwin|linux)-(x64|arm64)$/); + } + expect(new Set(tags).size, "duplicate tag in the pin table").toBe(tags.length); + }); + + // Microsoft publishes no osx-x86_64 asset from 1.27 on. main() has a branch that + // explains that and exits 0 so the Intel release build still succeeds; if someone + // adds a darwin-x64 entry, that branch makes it dead and the effect stays off. + it("does not pin darwin-x64, which upstream does not publish", () => { + expect(tags).not.toContain("darwin-x64"); + expect(source).toMatch(/tag === "darwin-x64"/); + }); +}); diff --git a/technical-documentation/engineering/webcam-segmentation.md b/technical-documentation/engineering/webcam-segmentation.md index 882752a21..a799a87e5 100644 --- a/technical-documentation/engineering/webcam-segmentation.md +++ b/technical-documentation/engineering/webcam-segmentation.md @@ -79,12 +79,52 @@ not an alpha channel, so nothing has to survive a codec that cannot carry one. `RightPanes.tsx`). A visible setting that changes nothing is worse than an absent one; it comes back for a platform the moment that platform's capture lands. What Linux still needs is `capture_webcam_rgb` and `set_webcam_mask` against the `ReadbackRing` it already has. -- **ONNX Runtime is not staged.** The crate links `ort` with `load-dynamic`, so nothing is needed - to *build*; at runtime `ensureOnnxRuntimeOnPath` looks for the library next to the addon in - `electron/native/bin//`, the convention `whisper-stt` already uses. Until a CI job puts it - there — the `build-whisper-stt.yml` pattern, roughly 15 MB per platform — the effect stays off. - `onnxruntime-node` was considered and rejected: 296 MB unpacked, which would roughly triple the - installer for three platforms' worth of providers we do not use. +- **Intel macOS and Linux have no runtime staged**, so the effect stays off there — for + different reasons, both recorded under *Staging* below. + +## Staging the runtime + +`scripts/fetch-onnxruntime.mjs`, wired into `build:mac`, `build:win` and `build:win:store`. The +crate links `ort` with `load-dynamic`, so nothing is needed to *build*; at runtime +`ensureOnnxRuntimeOnPath` looks for the library next to the addon in `electron/native/bin//`, +the convention `whisper-stt` already uses. The script downloads the pinned upstream release, +verifies its SHA-256 **before** unpacking, checks the archive's own LICENSE really is MIT, and +lifts out exactly one file. + +**The version is not free to move.** `crates/Cargo.toml` gives `ort` the feature `api-27`, which is +the minimum minor version it accepts — below it `GetApi` returns null and `ort` *panics* rather +than erroring. `scripts/fetch-onnxruntime.test.mjs` cross-checks the two pins, in both directions, +so a bump on either side cannot land alone. That coupling crosses a language boundary and is +invisible in review; it is the only thing standing between an `ort` bump and a render thread dying +on the first frame with an effect. + +What it actually costs, measured on the 1.27.1 artifacts rather than estimated: + +| target | library | size | shipped | +|---|---|---:|---| +| win32-x64 | `onnxruntime.dll` | 15.4 MB | yes | +| darwin-arm64 | `libonnxruntime.dylib` | 38.5 MB | yes | +| linux-x64 | `libonnxruntime.so` | 23.7 MB | pinned, not shipped | +| darwin-x64 | — | — | **no upstream build** | + +Two things fall out of that table, and neither was the expectation: + +- **macOS costs 2.5x what Windows does**, not the "roughly 15 MB per platform" this document + previously assumed. `strip -x` takes the dylib from 38.5 MB to 23.7 MB, but stripping it would + mean shipping something other than the artifact the SHA-256 pin vouches for, which is the whole + point of pinning. Not done; noted in case the installer size ever forces the trade. +- **Intel Macs cannot have it at all.** Microsoft publishes no `osx-x86_64` or universal asset for + any release from 1.27 on — arm64 is the only macOS target. The x64 DMG therefore ships without + the library and the effect is simply absent there, degrading exactly as designed. Building it + from source is the only way round it, and that is an ffmpeg-macos-sized script for a shrinking + platform. + +Linux is pinned in the same table but deliberately not wired into `build:linux`: the back-end +carries the shader half only, so the library would be 23 MB of installer for a code path that +cannot run. Landing the Linux capture half makes it a one-line change. + +`onnxruntime-node` was considered and rejected: 296 MB unpacked, which would roughly triple the +installer for three platforms' worth of providers we do not use. ## The constraint that shapes any fix