Fix CUDA sanity check for EB >=5.4.0 by casparvl · Pull Request #288 · EESSI/software-layer-scripts

casparvl · 2026-08-12T14:02:47Z

Since EB 5.4.0, the build dependencies are no longer available during the sanity check phase. This breaks our CUDA sanity check, which relies on cuobjdump being available.

This PR tries to promote CUDA to a regular dep temporarily, hoping that this will make the CUDA sanity check pass. A test on CUDA-Samples 12.9 shows that this is NOT the case. My AI friend thinks it's because the fake module that is generated is being generated based on data structures that are already available, so modifying the existing self.cfg instance has no effect.

Anyway, this is at least a description of the issue,and a starting point for the solution...

… the sanity check phase. This breaks our CUDA sanity check, which relies on cuobjdump

… running

casparvl · 2026-08-12T21:33:37Z

It loads the module, but I guess a subshell with an isolated environment may be started? I'm not sure. It still doesn't find cuobjdump, that's for sure:

== Loading CUDA module 'CUDA/12.9.1', temporarily making
available for the (CUDA) sanity check
  >> file 'bin/deviceQuery' found: OK
  >> file 'bin/matrixMul' found: OK
  >> file 'bin/cudaOpenMP' found: OK
  >> loading modules:
CUDA-Samples/12.9-GCC-14.3.0-CUDA-12.9.1...
== ... (took 26 secs)
== FAILED: Installation ended unsuccessfully: Failed to
execute CUDA sanity check: cuobjdump not found
CUDA module must be loaded for sanity check (or cuobjdump
available in PATH) (took 4 mins 50 secs)

ocaisa · 2026-08-12T21:41:51Z

I knew this would bite us eventually easybuilders/easybuild-framework#4692 (comment)

ocaisa · 2026-08-12T21:45:37Z

Since it just needs to be in the PATH and you know the location, you could modify the default environment to include that location

casparvl · 2026-08-12T21:46:47Z

Ok, according to deepwiki indeed the sanity check does a module purge. load_fake_module(purge=True, ...).

Some hint on how to resolve this may be gotten from https://deepwiki.com/search/in-the-sanity-check-step-a-fak_317cc243-13e3-4627-a8ed-fbec38bfa715?mode=fast

casparvl · 2026-08-12T21:51:02Z

Explicitely adding the location of the bin dir from the CUDA module's prefix to PATH might work... it won't be affected by a module purge. Since we RPATH anyway, it should just work - as long as we can get to that binary. It's not the most robust solution maybe, but the fact that sanity_check itself so heavily modifies the environment means that changes in pre_sanitycheck_hook have a high change of not surviving until cuobjdump is called... (I'm not even certain for the PATH change, but I think that will survive...).

Maybe the most robust way is to do the module load I do now, then do a which cuobjdump, get the prefix for that in some temporary env var (A), then unload the module again, then add A to the PATH.

…icitely

…make sure that step doesn't undo it

casparvl · 2026-08-12T22:03:41Z

Ok, testing this now, but not sure I can report the result - someone else may have to pick this up...

ocaisa · 2026-08-14T14:29:37Z

I'll see if I can get this to work

ocaisa · 2026-08-14T16:15:35Z

This works now:

== sanity checking...
== Running pre-sanitycheck hook...
== Add location of cuobjdump (/cvmfs/software.eessi.io/versions/2025.06/software/linux/aarch64/neoverse_n1/accel/nvidia/cc70/software/CUDA/12.9.1/bin) to
initial environ PATH for CUDA sanity check
  >> file 'bin/m4' found: OK
  >> loading modules: M4/1.4.20-GCCcore-14.3.0...
  >> CUDA sanity check summary report:
  >> Number of CUDA files checked: 0
  >> Number of files missing one or more CUDA Compute Capabilities: 0
  >> Number of files with device code for more CUDA Compute Capabilities than requested: 0
  >> Number of files missing PTX code for the highest configured CUDA Compute Capability: 0
== Running post-sanitycheck hook...
== Restoring stored initial environ PATH after CUDA sanity check
== ... (took 1 secs)

ocaisa · 2026-08-14T16:23:37Z

bot: build repo:eessi.io-2023.06-software instance:eessi-bot-mc-aws for:arch=x86_64/amd/zen2
bot: build repo:eessi.io-2025.06-software instance:eessi-bot-mc-aws for:arch=x86_64/amd/zen2
bot: build repo:eessi.io-2026.06-software instance:eessi-bot-mc-aws for:arch=x86_64/amd/zen2

eessi-bot-aws · 2026-08-14T16:23:45Z

New job on instance eessi-bot-mc-aws for repository eessi.io-2023.06-software
Building on: amd-zen2
Building for: x86_64/amd/zen2
Job dir: /project/def-users/SHARED/jobs/2026.08/pr_288/186725

date	job status	comment
Aug 14 16:23:43 UTC 2026	submitted	job id `186725` awaits release by job manager
Aug 14 16:24:50 UTC 2026	released	job awaits launch by Slurm scheduler
Aug 14 16:33:01 UTC 2026	running	job `186725` is running
Aug 14 16:39:18 UTC 2026	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-186725.out` ✅ no message matching `FATAL:` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.* created!` Artefacts `eessi-2023.06-software-linux-x86_64-amd-zen2-17867251720.tar.zst` size: 0 MiB (32416 bytes) entries: 1 modules under 2023.06/software/linux/x86_64/amd/zen2/modules/all no module files in tarball software under 2023.06/software/linux/x86_64/amd/zen2/software no software packages in tarball reprod directories under 2023.06/software/linux/x86_64/amd/zen2/reprod no reprod directories in tarball other under 2023.06/software/linux/x86_64/amd/zen2 no other files in tarball
Aug 14 16:39:18 UTC 2026	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ OK ] ( 1/10) EESSI_LAMMPS_lj %device_type=cpu %module_name=LAMMPS/29Aug2024-foss-2023b-kokkos %scale=1_node /aeb2d9df @BotBuildTests:x86-64-zen2+default P: perf: 448.08 timesteps/s (r:0, l:None, u:None) [ OK ] ( 2/10) EESSI_LAMMPS_lj %device_type=cpu %module_name=LAMMPS/2Aug2023_update2-foss-2023a-kokkos %scale=1_node /04ff9ece @BotBuildTests:x86-64-zen2+default P: perf: 453.943 timesteps/s (r:0, l:None, u:None) [ OK ] ( 3/10) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_allreduce %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %scale=1_node %device_type=cpu /775175bf @BotBuildTests:x86-64-zen2+default P: latency: 2.85 us (r:0, l:None, u:None) [ OK ] ( 4/10) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_allreduce %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %scale=1_node %device_type=cpu /52707c40 @BotBuildTests:x86-64-zen2+default P: latency: 2.5 us (r:0, l:None, u:None) [ OK ] ( 5/10) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_alltoall %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %scale=1_node %device_type=cpu /b1aacda9 @BotBuildTests:x86-64-zen2+default P: latency: 5.65 us (r:0, l:None, u:None) [ OK ] ( 6/10) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_alltoall %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %scale=1_node %device_type=cpu /c6bad193 @BotBuildTests:x86-64-zen2+default P: latency: 5.85 us (r:0, l:None, u:None) [ OK ] ( 7/10) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_latency %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %scale=1_node /15cad6c4 @BotBuildTests:x86-64-zen2+default P: latency: 0.85 us (r:0, l:None, u:None) [ OK ] ( 8/10) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_latency %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %scale=1_node /6672deda @BotBuildTests:x86-64-zen2+default P: latency: 0.73 us (r:0, l:None, u:None) [ OK ] ( 9/10) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_bw %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %scale=1_node /2a9a47b1 @BotBuildTests:x86-64-zen2+default P: bandwidth: 6459.72 MB/s (r:0, l:None, u:None) [ OK ] (10/10) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_bw %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %scale=1_node /1b24ab8e @BotBuildTests:x86-64-zen2+default P: bandwidth: 6470.38 MB/s (r:0, l:None, u:None) [ PASSED ] Ran 10/10 test case(s) from 10 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-186725.out` ✅ no message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`

eessi-bot-aws · 2026-08-14T16:23:50Z

New job on instance eessi-bot-mc-aws for repository eessi.io-2025.06-software
Building on: amd-zen2
Building for: x86_64/amd/zen2
Job dir: /project/def-users/SHARED/jobs/2026.08/pr_288/186726

date	job status	comment
Aug 14 16:23:48 UTC 2026	submitted	job id `186726` awaits release by job manager
Aug 14 16:24:47 UTC 2026	released	job awaits launch by Slurm scheduler
Aug 14 16:31:57 UTC 2026	running	job `186726` is running
Aug 14 16:37:14 UTC 2026	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-186726.out` ✅ no message matching `FATAL:` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.* created!` Artefacts `eessi-2025.06-software-linux-x86_64-amd-zen2-17867252890.tar.zst` size: 0 MiB (32418 bytes) entries: 1 modules under 2025.06/software/linux/x86_64/amd/zen2/modules/all no module files in tarball software under 2025.06/software/linux/x86_64/amd/zen2/software no software packages in tarball reprod directories under 2025.06/software/linux/x86_64/amd/zen2/reprod no reprod directories in tarball other under 2025.06/software/linux/x86_64/amd/zen2 no other files in tarball
Aug 14 16:37:14 UTC 2026	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ OK ] (1/6) EESSI_LAMMPS_lj %device_type=cpu %module_name=LAMMPS/22Jul2025-foss-2024a-kokkos %scale=1_node /ade8cad7 @BotBuildTests:x86-64-zen2+default P: perf: 435.596 timesteps/s (r:0, l:None, u:None) [ OK ] (2/6) EESSI_LAMMPS_lj %device_type=cpu %module_name=LAMMPS/22Jul2025_update4-foss-2025b-kokkos %scale=1_node /e121eb9c @BotBuildTests:x86-64-zen2+default P: perf: 443.605 timesteps/s (r:0, l:None, u:None) [ OK ] (3/6) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_allreduce %module_name=OSU-Micro-Benchmarks/7.5-gompi-2025a %scale=1_node %device_type=cpu /e4bf9965 @BotBuildTests:x86-64-zen2+default P: latency: 1.41 us (r:0, l:None, u:None) [ OK ] (4/6) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_alltoall %module_name=OSU-Micro-Benchmarks/7.5-gompi-2025a %scale=1_node %device_type=cpu /3da4890b @BotBuildTests:x86-64-zen2+default P: latency: 2.07 us (r:0, l:None, u:None) [ OK ] (5/6) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_latency %module_name=OSU-Micro-Benchmarks/7.5-gompi-2025a %scale=1_node /3255009a @BotBuildTests:x86-64-zen2+default P: latency: 0.19 us (r:0, l:None, u:None) [ OK ] (6/6) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_bw %module_name=OSU-Micro-Benchmarks/7.5-gompi-2025a %scale=1_node /59f4b331 @BotBuildTests:x86-64-zen2+default P: bandwidth: 7867.27 MB/s (r:0, l:None, u:None) [ PASSED ] Ran 6/6 test case(s) from 6 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-186726.out` ✅ no message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`

eessi-bot-aws · 2026-08-14T16:23:55Z

New job on instance eessi-bot-mc-aws for repository eessi.io-2026.06-software
Building on: amd-zen2
Building for: x86_64/amd/zen2
Job dir: /project/def-users/SHARED/jobs/2026.08/pr_288/186727

date	job status	comment
Aug 14 16:23:54 UTC 2026	submitted	job id `186727` awaits release by job manager
Aug 14 16:24:45 UTC 2026	released	job awaits launch by Slurm scheduler
Aug 14 16:30:53 UTC 2026	running	job `186727` is running
Aug 14 16:33:02 UTC 2026	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-186727.out` ✅ no message matching `FATAL:` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.* created!` Artefacts `eessi-2026.06-software-linux-x86_64-amd-zen2-17867251320.tar.zst` size: 0 MiB (32376 bytes) entries: 1 modules under 2026.06/software/linux/x86_64/amd/zen2/modules/all no module files in tarball software under 2026.06/software/linux/x86_64/amd/zen2/software no software packages in tarball reprod directories under 2026.06/software/linux/x86_64/amd/zen2/reprod no reprod directories in tarball other under 2026.06/software/linux/x86_64/amd/zen2 no other files in tarball
Aug 14 16:33:02 UTC 2026	test result	😢 FAILURE (click triangle for details) Reason EESSI test suite was not run, test step itself failed to execute. Details ✅ job output file `slurm-186727.out` ❌ found message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`

ocaisa

I'm ok with this now, but I had a significant hand in the final version so I leave it to @laraPPr to review/deploy

Since EB 5.4.0, the build dependencies are no longer available during…

4cb2c03

… the sanity check phase. This breaks our CUDA sanity check, which relies on cuobjdump

casparvl marked this pull request as draft August 12, 2026 14:04

Simply load the module as part of the hook. Not tested yet, currently…

d4b9721

… running

Fix function description

c8bac14

Caspar van Leeuwen added 2 commits August 13, 2026 00:01

See if we can just get the cuobjdump dir, and add it to the PATH expl…

d90c39f

…icitely

Should probably modify the path before unloading the module, just to …

4cc8edc

…make sure that step doesn't undo it

ocaisa mentioned this pull request Aug 13, 2026

{2025.06}[foss/2024a] LAMMPS 22Jul2025 with CUDA EESSI/software-layer#1461

Open

4 tasks

ocaisa reviewed Aug 13, 2026

View reviewed changes

Comment thread eb_hooks.py Outdated

laraPPr reviewed Aug 13, 2026

View reviewed changes

Comment thread eb_hooks.py

laraPPr reviewed Aug 14, 2026

View reviewed changes

Comment thread eb_hooks.py Outdated

Need to modify self.inital_environ to make cuobjdump available

4a932f9

ocaisa marked this pull request as ready for review August 14, 2026 16:18

ocaisa added ready-to-deploy ready-to-review labels Aug 14, 2026

ocaisa requested a review from laraPPr August 14, 2026 16:47

ocaisa approved these changes Aug 14, 2026

View reviewed changes

Uh oh!

Conversation

casparvl commented Aug 12, 2026

Uh oh!

casparvl commented Aug 12, 2026

Uh oh!

ocaisa commented Aug 12, 2026

Uh oh!

ocaisa commented Aug 12, 2026

Uh oh!

casparvl commented Aug 12, 2026

Uh oh!

casparvl commented Aug 12, 2026

Uh oh!

casparvl commented Aug 12, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ocaisa commented Aug 14, 2026

Uh oh!

ocaisa commented Aug 14, 2026

Uh oh!

ocaisa commented Aug 14, 2026

Uh oh!

eessi-bot-aws Bot commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eessi-bot-aws Bot commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eessi-bot-aws Bot commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ocaisa left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

eessi-bot-aws Bot commented Aug 14, 2026 •

edited

Loading

eessi-bot-aws Bot commented Aug 14, 2026 •

edited

Loading

eessi-bot-aws Bot commented Aug 14, 2026 •

edited

Loading