Implement Lazy Cachi2 Proxy for on-demand local builds

[jiridanek]

Problem

While prefetch-all.sh guarantees offline reproducibility for hermetic builds, running it locally requires downloading gigabytes of dependencies (pip wheels, RPMs, NPM tarballs) upfront before podman build can even start. Even with architecture filtering, this upfront tax degrades the local developer experience.

Proposed Solution (Phase 2: Long-Term Local Dev)

Implement a Lazy Cachi2 Proxy Daemon. Instead of downloading everything upfront to cachi2/output/, we run a lightweight local userspace server that intercepts the container's file requests and fetches dependencies from the internet on-demand.

Architectural Strategy

1. The Transport Layer:
   • Option A (FUSE via Privileged Container): Run a FUSE daemon in a privileged container on the Podman Machine (Linux VM). This bypasses the need for developers to install macFUSE (and reboot into macOS Recovery Mode) while providing a 100% POSIX-compliant local mount to the build container.
   • Option B (Userspace NFS Server): Run an unprivileged Go NFS server (willscott/go-nfs) on the host. Podman mounts it directly via --mount type=volume,volume-opt=type=nfs.
2. Language: Go (e.g., hanwen/go-fuse or go-nfs) or Rust (fuser) are highly recommended for handling high-concurrency FUSE/NFS translation and HTTP streaming safely.
3. The VFS (Virtual File System): At startup, the daemon parses the lockfiles (pylock.toml, package-lock.json, rpms.lock.yaml) and builds an in-memory tree. stat() and readdir() calls return instantly using the pre-calculated sizes and checksums.
4. Demand-Driven Fetching (Blocking Open): We do not need to predict the dependency graph. When the container executes RUN pip install and issues an open() syscall for numpy.whl, the daemon blocks the thread, downloads the file to a persistent host cache, and then returns success. The container build simply sees a file that took a few seconds to open.
5. Concurrency (Multi-Build Support): To support concurrent podman build instances, each build gets its own lightweight proxy instance. They share a global host cache (e.g., /var/tmp/lazy_cachi2_store). We use advisory filesystem locking (flock) on the cache files to prevent redundant downloads across concurrent daemons, falling back to atomic renames for corruption safety.

Exceptions

RPMs: dnf requires repodata/ indices upfront and won't fetch lazily. The proxy will still download RPMs upfront to run createrepo_c, but it will hash the lockfile to cache the repodata/ directory. For 99% of local rebuilds, the lockfile hash won't change, resulting in an instant cache hit and bypassing the RPM prefetch entirely.

cc @coderabbitai

[jiridanek]

@coderabbitai Please review this issue. Here is a summary and an outline for the implementation:

Summary:
To vastly improve the local developer experience, we need to replace the upfront, monolithic prefetch download with a "Lazy Cachi2 Proxy". This proxy will act as a virtual filesystem, intercepting container file requests and fetching exactly what is needed on-demand from the internet, while still strictly enforcing the hermetic lockfiles.

Implementation Outline:

1. Daemon Core: Create a lightweight daemon in Go (hanwen/go-fuse or go-nfs) or Rust (fuser). It will expose either a userspace NFS server (easier cross-platform mounting) or a FUSE filesystem (via a privileged Podman container).
2. VFS Initialization: On startup, the daemon parses pylock.toml, package-lock.json, artifacts.lock.yaml, and rpms.lock.yaml to construct an in-memory Virtual File System. File metadata (stat, readdir) must be served instantly from memory.
3. Demand-Driven Fetching: Implement a blocking Open() call. When the build container attempts to open a file (e.g., numpy.whl), the daemon halts the thread, downloads the file from the upstream URL to a persistent host cache, and then returns success.
4. Concurrency Safety: Implement advisory filesystem locking (flock) on the host cache directory. This ensures that if multiple concurrent podman build instances (and thus multiple proxy daemons) request the same file simultaneously, they share the download safely without corruption.
5. Build Integration: Update the Makefile to spin up an instance of the proxy daemon for the specific component being built, mount it into the container, and tear it down upon completion.

[coderabbitai]

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Understand current prefetch/cachi2 setup
echo "=== Lock files found ==="
fd -e toml -e yaml -e json --full-path . | rg -i "lock|cachi|prefetch" | head -40

echo ""
echo "=== Makefile prefetch/cachi2 targets ==="
rg -n "cachi|prefetch|fuse|nfs" Makefile 2>/dev/null | head -40

echo ""
echo "=== Any existing cachi2/prefetch scripts ==="
fd -t f | rg -i "cachi|prefetch" | head -20

echo ""
echo "=== Lock file examples (first few lines of each type) ==="
fd -g "artifacts.lock.yaml" | head -3 | xargs -I{} sh -c 'echo "--- {} ---"; head -20 {}'
fd -g "rpms.lock.yaml" | head -3 | xargs -I{} sh -c 'echo "--- {} ---"; head -20 {}'
fd -g "pylock.toml" | head -3 | xargs -I{} sh -c 'echo "--- {} ---"; head -20 {}'

Repository: opendatahub-io/notebooks
Length of output: 15449

---

🏁 Script executed:

#!/bin/bash
echo "=== Top-level directory structure ==="
tree -L 2 -d . 2>/dev/null | head -60

echo ""
echo "=== Makefile prefetch/download/vendor sections ==="
grep -n -A5 -B2 "prefetch\|download\|vendor\|cachi" Makefile 2>/dev/null | head -80

echo ""
echo "=== Any Containerfile/Dockerfile references to cachi or prefetch ==="
fd -g "Containerfile*" -g "Dockerfile*" | xargs grep -l "cachi\|prefetch" 2>/dev/null | head -10

echo ""
echo "=== package-lock.json count ==="
fd -g "package-lock.json" | wc -l

echo ""
echo "=== pylock.toml count ==="
fd -g "pylock.toml" | wc -l

echo ""
echo "=== artifacts.lock.yaml count ==="
fd -g "artifacts.lock.yaml" | wc -l

Repository: opendatahub-io/notebooks
Length of output: 4205

---

@jiridanek — solid concept that addresses a real pain point. Here's a detailed technical review:

---

Security

[CRITICAL] Checksum verification before Open() returns — CWE-367 (TOCTOU)

The proposal says the daemon "downloads the file…then returns success," but it must explicitly verify the downloaded content against the lockfile checksum before returning the file descriptor. If the file is served from cache without re-verifying, a compromised or corrupted cache entry bypasses hermetic guarantees entirely. Correct flow must be:

download → sha256(content) == lockfile_checksum ? serve : EACCES

For subsequent cache hits, either re-hash on every Open() (expensive but safe) or store a .verified sentinel alongside the cached file written atomically only after verification passes.

[HIGH] Strict filename → URL resolution — CWE-706 / CWE-22

The VFS readdir entries must come exclusively from the lockfile. The Open() handler must do an exact-match dictionary lookup (filename → entry) — no prefix matching, no path traversal, no fallback. A build container must not be able to construct a path that resolves outside the declared lockfile entries.

[MEDIUM] Host cache directory integrity — CWE-426

The persistent cache directory on the host must be owned and writable only by the daemon UID. If it is world-writable (e.g., under /tmp), an attacker on the same host can pre-populate a cache entry with a malicious file before the daemon verifies it. Use a mode-700 directory under ~/.cache/lazy-cachi2/ keyed by sha256, not filename.

[MEDIUM] Privileged FUSE container — CWE-250

If FUSE is chosen, the privileged Podman container running the daemon has CAP_SYS_ADMIN. Prefer the NFS/userspace path for rootless environments. If FUSE is used, scope capabilities precisely (--cap-add SYS_ADMIN --cap-drop ALL).

---

Architecture & Design

Blocking Open() with no timeout

A stalled download (flaky network, upstream 429/503) will block the build container's thread indefinitely with no user-visible progress. The daemon needs:

• A per-request timeout (configurable, e.g., 5 min default)
• Progress reporting to stderr of the daemon (so make log shows what is being fetched)
• On timeout/error: return EIO — the build fails loudly rather than hanging

Lock file parser complexity is underestimated

Four distinct formats need correct parsers:

File	Complexity	Notes
artifacts.lock.yaml	Low	Flat list, download_url+checksum+filename
rpms.lock.yaml	Medium	Arch-keyed (arches[].arch); daemon must be told target arch at startup — mirrors what RPM_ARCH does in the Makefile
pylock.toml (PEP 751)	High	Each package has both sdist and multiple wheels[] entries with PEP 508 marker expressions; daemon must evaluate markers for the target platform/Python version to pick the correct wheel URL. Using an existing PEP 508 evaluator (e.g., packaging in Python, or porting the logic to Go/Rust) is strongly recommended.
package-lock.json	Medium	npm v2/v3 nested format; only needed for codeserver — worth confirming whether these npm packages actually need to flow through the proxy or if they are only used during the patch application step (in which case they may not need FUSE/NFS at all)

The uv.lock.d/ layout also means multiple pylock variants per component (pylock.cpu.toml, pylock.cuda.toml, pylock.rocm.toml). The daemon must accept a specific variant at startup or merge all of them.

Cache key design

The cache must be content-addressed, keyed by the lockfile checksum value (sha256), not by filename. The same filename (e.g., numpy-2.0.0-cp312-linux_x86_64.whl) can appear in multiple components' lockfiles pointing to the same URL — content-addressing gives safe deduplication for free.

~/.cache/lazy-cachi2/
  sha256/
    8f8f67fd.../   # content hash prefix
      data         # downloaded bytes
      .verified    # written atomically only after hash check passes

Daemon lifecycle & Makefile integration

The Makefile teardown must use a trap to guarantee cleanup on SIGINT/SIGTERM:

build-with-proxy:
    ./lazy-cachi2 --lockfiles ... --arch $(RPM_ARCH) --mount /cachi2/output &
    DAEMON_PID=$$!
    trap "kill $$DAEMON_PID; umount /cachi2/output" EXIT INT TERM
    # wait for mount readiness (poll or use a readiness socket)
    $(CONTAINER_ENGINE) build ... --volume ...

Without the trap, a Ctrl-C mid-build leaves an orphaned daemon and a stale NFS/FUSE mount that will cause the next make invocation to fail.

NFS mount readiness race

The build must not start until the NFS server (or FUSE mount) is confirmed ready. A readiness mechanism — either a TCP health-check socket the daemon signals, or polling stat on the mount point — is required.

flock scope

flock(2) advisory locks are per-open-file-description and only work within a single host's kernel. They will correctly serialize concurrent podman build processes on the same machine downloading the same artifact. This is the right primitive here — just document that cross-host scenarios (e.g., parallel CI workers on different VMs sharing NFS-backed cache) require a different coordination layer (e.g., a .lock file with atomic rename, or a dedicated lock service).

---

Minor Points

• The proposal mentions pylock.toml as a single file, but in this repo it lives at uv.lock.d/pylock.<variant>.toml — the daemon's --lockfiles argument should accept a list.
• Consider exposing the daemon's download log (filename, URL, bytes, duration) in a structured format (--log-format json) so it can be consumed by CI dashboards to identify slow packages.
• The VFS should expose a read-only filesystem; Open() with O_WRONLY/O_RDWR should return EROFS immediately without touching the cache.