Revise CircleCI caching strategies

Context:

Clean build without cache on nix docker image: build time 1h6 minutes https://app.circleci.com/pipelines/github/tlaurion/heads/2538/workflows/7e6f69f6-7568-4edb-b3cf-a5c59fdf94ef
Build with cache on nix docker image (rebuild to reuse cache): build time 1h8 minutes https://app.circleci.com/pipelines/github/tlaurion/heads/2538/workflows/c8d82efa-aa49-48b5-bf53-cffd1c22268b

Cache speeding up builds : fail.

Time to download cache layer: 20 minutes https://app.circleci.com/pipelines/github/tlaurion/heads/2538/workflows/c8d82efa-aa49-48b5-bf53-cffd1c22268b/jobs/45679
Attaching a workspace prior of build : 2-3 minutes https://app.circleci.com/pipelines/github/tlaurion/heads/2538/workflows/c8d82efa-aa49-48b5-bf53-cffd1c22268b/jobs/45682/parallel-runs/0/steps/0-101
Persisting build to next board being dependent: 5 minutes https://app.circleci.com/pipelines/github/tlaurion/heads/2538/workflows/c8d82efa-aa49-48b5-bf53-cffd1c22268b/jobs/45682/parallel-runs/0/steps/0-108
Save cache (useless since same cache on rebuild) : 13 minutes https://app.circleci.com/pipelines/github/tlaurion/heads/2538/workflows/c8d82efa-aa49-48b5-bf53-cffd1c22268b/jobs/45708

Basically we loose around 45 minutes, some of which blocks other boards from building.

Some thoughts:

We should build muslcross alone on circleci and chache and reuse that.
For the rest i'm not sure I see the point of using caching anymore for to save CPU time from CI at the cost of more bandwidth and waiting time prior of starting builds in most use case
On nix, we could put musl-cross-make as first step for both x86 and ppc64 under docker image and modify Makefile to check if provided per nix and if the case use it
advance on prior step and use nix coreboot-buildstacks

After that, optimizations would come from moving all modules into nix expressions. But I'm not sure we want to go there yet. Seems to close contributions to nix developers which I do not categorize myself in still.

As confirmed under #1967

As said under #1666, we need insights on best next strategies to save cache/pass workspace caches under CircleCI per above links. Short version:

* No cache https://app.circleci.com/pipelines/github/tlaurion/heads/3315/workflows/9bc8a009-68c8-4547-9232-3ba3612ab0fb
  
  * restore cache step (prep_env) : 1m16s : no cache downloaded nor applied
  * musl-cross-make built from source: 16m36s
  * **total pipeline time: 1h24m (most of this time in compute time : no cache reused)**

* First layer cache reusal (msul-cross-make) https://app.circleci.com/pipelines/github/tlaurion/heads/3314/workflows/e588480d-d13a-49e0-a1b6-78fed839b70b
  
  * restore cache step (prep_env) : 9m2s
  * musl-cross-make cache reused step time: 11m43s : https://app.circleci.com/pipelines/github/tlaurion/heads/3314/workflows/e588480d-d13a-49e0-a1b6-78fed839b70b/jobs/67601
    
    * build time is still of 5m44s, rest is workspace download from prep_env step (attaching workspace step of 1m20s) and workspace creation (persisting to workspace step of  3m40s
    * **total build time : 1h24m29s (most of this time in computer time still for board builds but less cpu used because musl reused for ppc64/x86)**

* third layer cache reusal (all built artifacts downloaded to minimize CPU build time) https://app.circleci.com/pipelines/github/tlaurion/heads/3315/workflows/c66c8669-327a-42d9-822a-47b2dc87241e
  
  * restore cache step (prep_env) : 16m9s
  * **t480-hotp-maximized : 18m30s here vs 42m6s: but only one board cache is used by arch right now**
  * total build time : 1h43m30s : we basically try to be more gentle for CircleCI CPU used time but do not economize much here and our board build time is less efficient than if we were not reusing any cache
    
    * To be efficient, we would need to:
      
      * Have musl-cross-make built into docker image for architectures we support instead of rebuilding from source, and/or
      * Combine workspace cache layers per arch, then have save_Cache depend on that layer so we have the same gains of the single cahced board build layer we currently have but for all boards (this is currently 50% less build time whe board cache layer is used for talos-2 and t480, which layer is reused only for board depending on it, otherwise twice time spent in building from source all coreboot and modules

way improved under #2039

Navigation Menu

Repository navigation

Revise CircleCI caching strategies #1666

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Revise CircleCI caching strategies #1666

Description

Activity

tlaurion commented on May 5, 2025

tlaurion commented on Apr 20, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions