Update site package behavior#232
Conversation
…SOFTWARE_PREFIX, rather than only supporting the default under host_injections
| -- If EESSI_SITE_SOFTWARE_PREFIX is defined, replace /cvmfs/software.eessi.io (or more generally EESSI_CVMFS_REPO) | ||
| -- by that prefix. This ensures that the directory still contains the os/vendor/arch/micro-arch/accelerator etc | ||
| -- If it is not defined, default to a site installation prefix under host_injections | ||
| site_prefix = os.getenv("EESSI_SITE_SOFTWARE_PREFIX") |
There was a problem hiding this comment.
I don't love this approach, there is value in having the path in a fully defined location.
There was a problem hiding this comment.
I thought about it more, and I am coming around. I was a bit stuck on having the fixed path because we require a fixed path to be able to do MPI injection. However, these are actually separate issues and this change doesn't affect that.
There was a problem hiding this comment.
This will require some documentation updates, and some additional checks in EESSI-extend. We should also update the dev.eessi.io workflow to use this.
Issues for all these are enough for now, but they should be addressed.
There was a problem hiding this comment.
Caspar and I discussed this quite a bit yesterday. In some way I also still prefer to have everything in a single and fixed location, but I also see Caspar's point, and being able to do the software installations elsewhere does make sense to me. I still have some concerns, e.g. about where to put the Lmod site hooks: Caspar wanted to keep them under host_injections, as they (may) also affect EESSI modules, and though that makes sense, it does mean that you may have two locations with this 2023.06/software/x86_64/amd/zen3 trees (one for locally built software, one for Lmod hooks). Same question for the site's Lmod cache: for that one it does definitely make sense to store them near the software, meaning you will end up with two of those .lmod directories in different places. Personally, I still don't really like that, it can easily confuse users/admins.
The current situation, with just a single host_injection directory, also has some drawbacks, e.g. in case you want to do site installations on a local CVMFS repo, while having MPI/GPU libraries on a local disk. You can do it by doing some symlink trickery, but that's not ideal either.
Anyway, since the default behavior won't change, I was okay with adding the possibility for overriding the software installation prefix.
| -- Make sure the EESSI cache is found, this is specified in the lmodrc.lua in the eessi_software_path | ||
| prepend_path("LMOD_RC", pathJoin(eessi_software_path, ".lmod", "lmodrc.lua")) | ||
| eessiDebug("Adding " .. pathJoin(eessi_software_path, ".lmod", "lmodrc.lua") .. " to LMOD_RC") | ||
| -- Make sure that a cache for site installations can also be found |
There was a problem hiding this comment.
How to generate the cache also needs an update, but I need to check that works first. We need to let Lmod know that the module path requires a gateway module, then the hierarchy can be represented in the cache.
It's a separate issue, this just reminds me that I need to look into it.
There was a problem hiding this comment.
Do we currently even support caches for local installation? Caspar and I looked into that a bit yesterday, but we couldn't find anything. We plan to cover this in the tutorial, but we should indeed also add documentation for this. With Caspar's proposed change, sites can easily call the create_lmodrc.py from our repo to generate the lmodrc.lua file for their local stack (the EESSI module will then add it to $LMOD_RC), and then it should just be a matter of calling for instance https://github.com/EESSI/filesystem-layer/blob/main/scripts/update_lmod_caches.sh to actually generate the cache files.
3 participants
This is preparatory work for better supporting building on top of EESSI.
Specifically, this change allows sites to customize where their site installation path is through
EESSI_SITE_SOFTWARE_PREFIX. It also makes sure that any caches for the site-installs are picked up by adding thelmodrcfile to theLDOD_RCsearch path. Thus, as long as sites generate anlmodrc.lua(e.g. using thecreate_lmodrc.pyfrom thesoftware-layer-scriptsrepo) and run the appropriate commands to generate a cache in the corresponding.lmodfolder (analogous to how we do this for EESSI), this will allow them to also have a proper cache for locally installed modules.An example would be: