/
Using available software - Environment Modules

Using available software - Environment Modules

Owned by Robert Schulz
Last updated: 14/05/2024
10 min read

This section explains how to make use of the system-wide available software stack which is organized in modules.

A module is a self-contained description of a software package - it contains the settings required to run a software package and, usually, encodes required dependencies on other software packages.

There are a number of different environment module implementations commonly used on HPC systems: the two most common are TCL modules and Lmod. Both of these use similar syntax and the concepts are the same so learning to use one will allow you to use whichever is installed on the system you are using.

Both ALICE and SHARK, use Lmod.

The module system makes it easy to switch between different software for different jobs and access multiple versions of the same software.

Basic usage

This table provides a short overview of the most commonly used options for the command module. Check the help for module to see all commands with a description.

Command

Description

Command

Description

module -h or module --help

Get help message

module list

Get a list of all currently loaded modules

module avail or module av

Get list of all available modules (See below for details)

module avail <string> or module av <string>

Get list of all available modules which contain <string> in their name (See below for details)

module spider <string>

Get list of all modules matching <string> and their description

module whatis <module>

Get the description of the given <module>

module load <module>

Load the specified <module> (See below for details)

module unload <module>

Unload the module <module> (See below for details)

module reset

Reset module environment to default list of modules (See below for details)

module purge

Unload all loaded modules (See below for details)

module save

Save the current list of modules, e.g., as a default environment

module restore

Restore module environment to a default list of modules

The module structure on ALICE and SHARK

ALICE and SHARK have different module structures. This is partly related to how the system-wide software stack is being managed on both clusters. However, the basic usage of environment modules is the same on both clusters.

Once you are familiar with how modules are organized on the cluster that you are using, it will not be difficult for you to find the modules that you are looking for.

ALICE

On ALICE modules are organized as different module trees. The software behind each module is build and maintained using EasyBuild (https://easybuild.io/).

In each tree, the modules are organized as <name>/<version> where <version> can also include additional information about what was used to build the module <name>

The default trees are

Tree

Description

Tree

Description

/easybuild/modules/all

 

Default tree, containing all of the system-wide software build with EasyBuild for all nodes with Intel CPU. Available by default or after loading ALICE/default module.

/cm/shared/modulefiles

Containing the modules for Slurm and LUMC/FSL software

Additionally available trees that need to be loaded manually are:

Tree

Description

Tree

Description

/cm/shared/LUMC/modulefiles

Containing all LUMC-provided modules including FSL. Available after loading the module LUMC

/cm/shared/easybuild/GenuineIntel/modules/all

and
/cm/shared/easybuild/AuthenticAMD/modules/all

Old modules for Intel and AMD nodes prior to maintenance of May 2024. Because they were build on the old operating system they may or may not work. You can access them by loading the ALICE/legacy module. The module tree is chosen based on the CPU type.

SHARK

On SHARK, all software is contained in the tree /share/modulefiles

Modules are being organized in a hierarchical structure, i.e., <moduleclass>/<name>/<version> where <moduleclass> is the class to which the module belong to, <name> the name of the module, and <version> the installed version of the module <name>

The different module classes currently are:

  • benchmark

  • bioimage

  • bioinformatics

  • container

  • cryogenicEM

  • genomics

  • graphics

  • gwas

  • library

  • mathematical

  • medicalImaging

  • neuroImaging

  • pharmaceutical

  • statistical

  • system

  • tools

List currently loaded modules

On login, you might start out with a default set of modules.

The module list command shows which modules you currently have loaded in your environment.

ALICE

This is what you will most likely see after logging into ALICE.

[me@nodelogin02~]$ module list Currently Loaded Modules: 1) shared 2) DefaultModules 3) gcc/8.2.0 4) slurm/19.05.1

You can see that by default the module for Slurm and the gcc compiler is loaded. The number behind the slash sign represent the version number.

SHARK

On SHARK, no modules are being loaded by default. After login, you would see

[me@res-hpc-lo02 ~]$ module list No modules loaded

List available modules

To see the available modules, use module -d avail. With the -d option, you will only get the default versions of the modules. For various software packages, there are also older/other versions available, that might be used if necessary. You can see all version by omitting the -d option.

ALICE

[me@nodelogin02~]$ module -d avail ---------------------------------------------------------------------- /cm/shared/easybuild/GenuineIntel/modules/all ---------------------------------------------------------------------- 4ti2/1.6.9-GCC-8.2.0-2.31.1 Miniconda2/4.7.10 flatbuffers/1.12.0-GCCcore-10.2.0 ALICE/default (L) Miniconda3/4.9.2 flex/2.6.4 AMUSE-GPU/12.0.0-foss-2018a-Python-2.7.14 MotionCor2/1.3.0-GCCcore-8.2.0 fontconfig/2.13.93-GCCcore-10.3.0 AMUSE-Miniconda2/4.7.10 MrBayes/3.2.6-foss-2017a foss/2021a AMUSE-VADER/12.0.0-foss-2018a-Python-2.7.14 MultiQC/1.7-foss-2018b-Python-3.6.6 fosscuda/2020b AMUSE/2021.7-Miniconda3-4.9.2 NASM/2.15.05-GCCcore-10.3.0 freeglut/3.2.1-GCCcore-8.3.0 ANTs/2.3.5-foss-2019b-Python-3.7.4 NCCL/2.10.3-GCCcore-10.3.0-CUDA-11.3.1 freetype/2.10.4-GCCcore-10.3.0 APR-util/1.6.1-GCCcore-8.2.0 NGS/2.10.0-GCCcore-8.2.0-Java-11 gc/7.6.12-GCCcore-9.3.0 APR/1.7.0-GCCcore-8.2.0 NLTK/3.2.4-foss-2019a-Python-3.7.2 gcccuda/2020b ARPwARP/8.0 NLopt/2.6.2-GCCcore-10.2.0 gensim/3.7.3-foss-2019a-Python-3.7.2 # etc...

SHARK

[rfschulz@res-hpc-lo02 ~]$ module -d avail ----------------------------------------------------------------------------------- /share/modulefiles ------------------------------------------------------------------------------------ benckmarks/disk/ior/3.4.0/gcc-8.3.1 library/cuda/11.0/gcc.8.3.1 neuroImaging/MRtrix3/3.0.3/gcc-8.3.1 benckmarks/disk/iozone/3.491/gcc-8.3.1 library/cuda/11.1/gcc.8.3.1 neuroImaging/SimpleElastix/0.10.0/python3.6.8 bioimage/CellProfiler/4.2.1/gcc-8.3.1 library/cuda/11.2/gcc.8.3.1 neuroImaging/dsi-studio/2021.10.05/gcc-8.3.1 bioimage/QuPath/0.2.3 library/cuda/11.3/gcc.8.3.1 neuroImaging/freesurfer/7.2.0 bioimage/bftools/6.8.1 library/cuda/11.4.1/gcc.8.3.1 neuroImaging/fsl/5.0.10 bioimage/slideToolkit/1.1 library/cuda/11.6.1/gcc.8.3.1 neuroImaging/fsl/5.0.11 bioinformatics/tools/SNP/flashpca/2.1/gcc-8.3.1 library/cuda/7.5/gcc.8.3.1 neuroImaging/fsl/5.0.9 bioinformatics/tools/assembly-stats/1.0.1/gcc-8.3.1 library/cuda/8.0/gcc.8.3.1 neuroImaging/fsl/6.0.0 bioinformatics/tools/ncbi/sra/2.10.9 library/cuda/9.0/gcc.8.3.1 neuroImaging/fsl/6.0.1 container/singularity/3.10.0/gcc.8.5.0 library/cuda/9.1/gcc.8.3.1 neuroImaging/fsl/6.0.2 # etc...

Finding specific modules

If you are searching for a specific software package or tool you can search for the full module name like this:

ALICE

[me@nodelogin02~]$ module avail python ---------------------------------------- /cm/shared/easybuild/modules/all ----------------------------------------- AMUSE-GPU/12.0.0-foss-2018a-Python-2.7.14 AMUSE-VADER/12.0.0-foss-2018a-Python-2.7.14 AMUSE/12.0.0-foss-2018a-Python-2.7.14 AMUSE/13.1.0-foss-2018a-Python-3.6.4 (D) Biopython/1.71-foss-2018a-Python-2.7.14 Biopython/1.73-foss-2019a Biopython/1.75-foss-2019b-Python-3.7.4 (D) Boost.Python/1.67.0-foss-2018b-Python-3.6.6 CGAL/4.11.1-foss-2018b-Python-3.6.6 CONCOCT/1.1.0-foss-2019a-Python-2.7.15 CheckM/1.0.18-foss-2019a-Python-2.7.15 Cython/0.25.2-foss-2018a-Python-2.7.14 Cython/0.25.2-foss-2018a-Python-3.6.4 Cython/0.29.3-foss-2019a-Python-3.7.2 (D) DAS_Tool/1.1.1-foss-2019a-R-3.6.0-Python-3.7.2 Docutils/0.9.1-foss-2018a-Python-2.7.14 Docutils/0.9.1-foss-2018a-Python-3.6.4 (D) GObject-Introspection/1.60.1-GCCcore-8.2.0-Python-3.7.2 IPython/5.7.0-foss-2018a-Python-2.7.14 IPython/6.4.0-foss-2018a-Python-3.6.4 IPython/7.7.0-foss-2019a-Python-3.7.2 (D) Keras/2.2.4-foss-2019a-Python-3.7.2 Mako/1.0.7-foss-2017b-Python-2.7.14 Mako/1.0.7-foss-2018a-Python-2.7.14 Mako/1.0.7-foss-2018b-Python-2.7.15 Meson/0.50.0-GCCcore-8.2.0-Python-3.7.2 Meson/0.51.2-GCCcore-8.3.0-Python-3.7.4 (D) MultiQC/1.7-foss-2018b-Python-3.6.6 NLTK/3.2.4-foss-2018a-Python-3.6.4 NLTK/3.2.4-foss-2019a-Python-3.7.2 (D) PyCairo/1.18.0-foss-2018b-Python-3.6.6 PyTorch/1.3.1-fosscuda-2019b-Python-3.7.4 PyYAML/3.13-foss-2018b-Python-3.6.6 Python/2.7.14-foss-2017b Python/2.7.14-foss-2018a Python/2.7.14-GCCcore-6.4.0-bare Python/2.7.15-foss-2018b Python/2.7.15-GCCcore-7.3.0-bare Python/2.7.15-GCCcore-8.2.0 Python/2.7.16-GCCcore-8.3.0 Python/3.6.4-foss-2018a Python/3.6.6-foss-2018b Python/3.7.2-GCCcore-8.2.0 Python/3.7.4-GCCcore-8.3.0 (D) etc etc etc .......

To only search for the module Python itself, use

module -r avail '^Python/'

SHARK

me@res-hpc-lo02 ~]$ module avail python --------------------- /share/modulefiles ---------------------- genomics/ngs/tools/liftoff/1.5.1/python-3.8 library/wxpython/4.1.0/gcc-8.3.1 library/wxpython/4.1.1/gcc-8.3.1 neuroImaging/SimpleElastix/0.10.0/python3.6.8 system/python/2.7.17 system/python/3.7.6 system/python/3.8.1 system/python/3.9.12 system/python/3.9.13 system/python/3.10.2 (L,D) tools/miniconda/python2.7/4.7.12 tools/miniconda/python3.8/4.8.5 tools/miniconda/python3.8/4.9.2 (D) tools/miniconda/python3.9/4.12.0

In the above output, you can see that there are modules with the flag "(D)". This indicates that this is the default module for a software package for which modules for different versions exists.

If you want to get more information about a specific module, you can use the whatis sub-command, e.g.

ALICE

[me@nodelogin02 ~]$ module whatis Python/3.7.4-GCCcore-8.3.0

SHARK

[me@res-hpc-lo02 ~]$ module whatis system/python/3.10.2

Load modules

To load a module, use module load.

In the example below, we will use R to demonstrate how the module system works and what it does to your user environment.

Initially, R is not loaded and therefore the command Rscript is not available for use. We can test this by using the command which that looks for programs the same way that Bash does. We can use it to tell us where a particular piece of software is stored.

ALICE

[me@nodelogin02 ~]$ which Rscript /usr/bin/which: no Rscript in (/cm/shared/apps/slurm/20.11.9/sbin:/cm/shared/apps/slurm/20.11.9/bin:/cm/local/apps/gcc/8.2.0/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/me/.local/bin:/home/me/bin)

SHARK

[me@res-hpc-lo02 ~]$ which Rscript /usr/bin/which: no Rscript in (/share/software/system/python/3.10.2/bin:/home/me/.local/bin:/home/me/bin:/opt/TurboVNC/bin:/opt/VirtualGL/bin:/share/software/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/me/.dotnet/tools)

The list of paths in the brackets after “no Rscript in” comes from the environment variable $PATH.

Let us first find the a module for R which contains the Rscript command

ALICE

[me@nodelogin02 ~]$ module -r avail '^R/' ----------------------------------------------------------------------------- /cm/shared/easybuild/modules/all ----------------------------------------------------------------------------- R/3.6.0-foss-2019a-Python-3.7.2 R/3.6.0-foss-2019a R/3.6.2-foss-2019b R/3.6.2-fosscuda-2019b R/4.0.0-foss-2020a R/4.0.5-foss-2020b (D)

SHARK

[me@res-hpc-lo02 ~]$ module -r avail '/R/' ------------------------------------------------------------------------------------ /share/modulefiles ------------------------------------------------------------------------------------ statistical/R/3.4.4/gcc.8.3.1 statistical/R/3.6.0/gcc.8.3.1 statistical/R/3.6.3/gcc.8.3.1 statistical/R/4.1.0/gcc.8.3.1 statistical/R/3.5.3/gcc.8.3.1 statistical/R/3.6.2/gcc.8.3.1 statistical/R/4.0.2/gcc.8.3.1 statistical/R/4.1.2/gcc.8.3.1

Now that we have found the module, we can load it, list all loaded modules just to make sure and check whether Rscript is now available:

ALICE

[schulzrf@nodelogin02 ~]$ module load R/4.0.5-foss-2020b [schulzrf@nodelogin02 ~]$ module list Currently Loaded Modules: 1) shared 23) foss/2020b 45) gettext/0.21-GCCcore-10.2.0 67) Szip/2.1.1-GCCcore-10.2.0 2) DefaultModules 24) bzip2/1.0.8-GCCcore-10.2.0 46) PCRE/8.44-GCCcore-10.2.0 68) HDF5/1.10.7-gompi-2020b 3) gcc/8.2.0 25) expat/2.2.9-GCCcore-10.2.0 47) GLib/2.66.1-GCCcore-10.2.0 69) UDUNITS/2.2.26-GCCcore-10.2.0 4) slurm/20.11.9 26) libpng/1.6.37-GCCcore-10.2.0 48) cairo/1.16.0-GCCcore-10.2.0 70) GSL/2.6-GCC-10.2.0 5) GCCcore/10.2.0 27) Brotli/1.0.9-GCCcore-10.2.0 49) libreadline/8.0-GCCcore-10.2.0 71) Ghostscript/9.53.3-GCCcore-10.2.0 6) zlib/1.2.11-GCCcore-10.2.0 28) freetype/2.10.3-GCCcore-10.2.0 50) Tcl/8.6.10-GCCcore-10.2.0 72) JasPer/2.0.24-GCCcore-10.2.0 7) binutils/2.35-GCCcore-10.2.0 29) ncurses/6.2-GCCcore-10.2.0 51) SQLite/3.33.0-GCCcore-10.2.0 73) LittleCMS/2.11-GCCcore-10.2.0 8) GCC/10.2.0 30) util-linux/2.36-GCCcore-10.2.0 52) PCRE2/10.35-GCCcore-10.2.0 74) ImageMagick/7.0.10-35-GCCcore-10.2.0 9) numactl/2.0.13-GCCcore-10.2.0 31) fontconfig/2.13.92-GCCcore-10.2.0 53) NASM/2.15.05-GCCcore-10.2.0 75) GLPK/4.65-GCCcore-10.2.0 10) XZ/5.2.5-GCCcore-10.2.0 32) xorg-macros/1.19.2-GCCcore-10.2.0 54) libjpeg-turbo/2.0.5-GCCcore-10.2.0 76) nodejs/12.19.0-GCCcore-10.2.0 11) libxml2/2.9.10-GCCcore-10.2.0 33) X11/20201008-GCCcore-10.2.0 55) jbigkit/2.1-GCCcore-10.2.0 77) Python/3.8.6-GCCcore-10.2.0 12) libpciaccess/0.16-GCCcore-10.2.0 34) gzip/1.10-GCCcore-10.2.0 56) LibTIFF/4.1.0-GCCcore-10.2.0 78) netCDF/4.7.4-gompi-2020b 13) hwloc/2.2.0-GCCcore-10.2.0 35) lz4/1.9.2-GCCcore-10.2.0 57) Java/11.0.2 79) GEOS/3.9.1-GCC-10.2.0 14) libevent/2.1.12-GCCcore-10.2.0 36) zstd/1.4.5-GCCcore-10.2.0 58) Tk/8.6.10-GCCcore-10.2.0 80) PROJ/7.2.1-GCCcore-10.2.0 15) UCX/1.9.0-GCCcore-10.2.0 37) libdrm/2.4.102-GCCcore-10.2.0 59) cURL/7.72.0-GCCcore-10.2.0 81) libgeotiff/1.6.0-GCCcore-10.2.0 16) libfabric/1.11.0-GCCcore-10.2.0 38) libglvnd/1.3.2-GCCcore-10.2.0 60) GMP/6.2.0-GCCcore-10.2.0 82) pybind11/2.6.0-GCCcore-10.2.0 17) PMIx/3.1.5-GCCcore-10.2.0 39) libunwind/1.4.0-GCCcore-10.2.0 61) NLopt/2.6.2-GCCcore-10.2.0 83) SciPy-bundle/2020.11-foss-2020b 18) OpenMPI/4.0.5-GCC-10.2.0 40) LLVM/11.0.0-GCCcore-10.2.0 62) libogg/1.3.4-GCCcore-10.2.0 84) libtirpc/1.3.1-GCCcore-10.2.0 19) OpenBLAS/0.3.12-GCC-10.2.0 41) Mesa/20.2.1-GCCcore-10.2.0 63) FLAC/1.3.3-GCCcore-10.2.0 85) HDF/4.2.15-GCCcore-10.2.0 20) gompi/2020b 42) libGLU/9.0.1-GCCcore-10.2.0 64) libvorbis/1.3.7-GCCcore-10.2.0 86) GDAL/3.2.1-foss-2020b 21) FFTW/3.3.8-gompi-2020b 43) pixman/0.40.0-GCCcore-10.2.0 65) libsndfile/1.0.28-GCCcore-10.2.0 87) R/4.0.5-foss-2020b 22) ScaLAPACK/2.1.0-gompi-2020b 44) libffi/3.3-GCCcore-10.2.0 66) ICU/67.1-GCCcore-10.2.0 [me@nodelogin02 ~]$ which Rscript /cm/shared/easybuild/GenuineIntel/software/R/4.0.5-foss-2020b/bin/Rscript

While you can also just use module load R, this is not recommended because this command will always load the default version of a module and the default may change when newer versions are being added.

On ALICE, the long list of modules is completely fine because the module R depends on several other modules which have to be loaded in order for R to properly work.

SHARK

[me@res-hpc-lo02 ~]$ module load statistical/R/4.1.2 [me@res-hpc-lo02 ~]$ module list Currently Loaded Modules: statistical/R/4.1.2/gcc.8.3.1 [em@res-hpc-lo02 ~]$ which Rscript /share/software/statistical/R/R-4.1.2/bin/Rscript

So what just happened? To understand the output, first we need to understand the nature of the $PATH environment variable. $PATH is a special environment variable that controls where a Linux operating system (OS) looks for software. Specifically $PATH is a list of directories (separated by :) that the OS searches through for a command. As with all environment variables, we can print it using echo.

ALICE

This is how $PATH looks like after loading R on ALICE

[me@nodelogin02 ~]$ echo $PATH /cm/shared/easybuild/GenuineIntel/software/R/4.0.5-foss-2020b/bin:/cm/shared/easybuild/GenuineIntel/software/GDAL/3.2.1-foss-2020b/bin:/cm/shared/easybuild/GenuineIntel/software/HDF/4.2.15-GCCcore-10.2.0/bin:/cm/shared/easybuild/GenuineIntel/software/SciPy-bundle/2020.11-foss-2020b/bin:/cm/shared/easybuild/GenuineIntel/software/pybind11/2.6.0-GCCcore-10.2.0/bin:/cm/shared/easybuild/GenuineIntel/software/libgeotiff/1.6.0-GCCcore-10.2.0/bin:/cm/shared/easybuild/GenuineIntel/software/PROJ/7.2.1-GCCcore-10.2.0/bin:/cm/shared/easybuild/GenuineIntel/software/GEOS/3.9.1-GCC-10.2.0/bin:/cm/shared/easybuild/GenuineIntel/software/netCDF/4.7.4-gompi-2020b/bin:/cm/shared/easybuild/GenuineIntel/software/Python/3.8.6-GCCcore-10.2.0/bin:/cm/shared/easybuild/GenuineIntel/software/nodejs/12.19.0-GCCcore-10.2.0/bin:/cm/shared/easybuild/GenuineIntel/software/GLPK/4.65-GCCcore-10.2.0/bin:/cm/shared/easybuild/GenuineIntel/software/ImageMagick/7.0.10-35-GCCcore-10.2.0/bin:/cm/shared/easybuild/GenuineIntel/software/LittleCMS/2.11-GCCcore-10.2.0/bin:/cm/shared/easybuild/GenuineIntel/software/JasPer/2.0.24-GCCcore-10.2.0/bin:/cm/shared/easybuild/GenuineIntel/software/Ghostscript/9.53.3-GCCcore-10.2.0/bin:/cm/shared/easybuild/GenuineIntel/software/GSL/2.6-GCC-10.2.0/bin:/cm/shared/easybuild/GenuineIntel/software/UDUNITS/2.2.26-GCCcore-10.2.0/bin:/cm/shared/easybuild/GenuineIntel/software/HDF5/1.10.7-gompi-2020b/bin:/cm/shared/easybuild/GenuineIntel/software/ICU/67.1-GCCcore-10.2.0/sbin:/cm/shared/easybuild/GenuineIntel/software/ICU/67.1-GCCcore-10.2.0/bin:/cm/shared/easybuild/GenuineIntel/software/libsndfile/1.0.28-GCCcore-10.2.0/bin:/cm/shared/easybuild/GenuineIntel/software/FLAC/1.3.3-GCCcore-10.2.0/bin:/cm/shared/easybuild/GenuineIntel/software/cURL/7.72.0-GCCcore-10.2.0/bin:/cm/shared/easybuild/GenuineIntel/software/Tk/8.6.10-GCCcore-10.2.0/bin:/cm/shared/easybuild/software/Java/11.0.2:/cm/shared/easybuild/software/Java/11.0.2/bin:/cm/shared/easybuild/GenuineIntel/software/LibTIFF/4.1.0-GCCcore-10.2.0/bin:/cm/shared/easybuild/GenuineIntel/software/jbigkit/2.1-GCCcore-10.2.0/bin:/cm/shared/easybuild/GenuineIntel/software/libjpeg-turbo/2.0.5-GCCcore-10.2.0/bin:/cm/shared/easybuild/GenuineIntel/software/NASM/2.15.05-GCCcore-10.2.0/bin:/cm/shared/easybuild/GenuineIntel/software/PCRE2/10.35-GCCcore-10.2.0/bin:/cm/shared/easybuild/GenuineIntel/software/SQLite/3.33.0-GCCcore-10.2.0/bin:/cm/shared/easybuild/GenuineIntel/software/Tcl/8.6.10-GCCcore-10.2.0/bin:/cm/shared/easybuild/GenuineIntel/software/cairo/1.16.0-GCCcore-10.2.0/bin:/cm/shared/easybuild/GenuineIntel/software/GLib/2.66.1-GCCcore-10.2.0/bin:/cm/shared/easybuild/GenuineIntel/software/PCRE/8.44-GCCcore-10.2.0/bin:/cm/shared/easybuild/GenuineIntel/software/gettext/0.21-GCCcore-10.2.0/bin:/cm/shared/easybuild/GenuineIntel/software/LLVM/11.0.0-GCCcore-10.2.0/bin:/cm/shared/easybuild/GenuineIntel/software/zstd/1.4.5-GCCcore-10.2.0/bin:/cm/shared/easybuild/GenuineIntel/software/lz4/1.9.2-GCCcore-10.2.0/bin:/cm/shared/easybuild/GenuineIntel/software/gzip/1.10-GCCcore-10.2.0/bin:/cm/shared/easybuild/GenuineIntel/software/X11/20201008-GCCcore-10.2.0/bin:/cm/shared/easybuild/GenuineIntel/software/fontconfig/2.13.92-GCCcore-10.2.0/bin:/cm/shared/easybuild/GenuineIntel/software/util-linux/2.36-GCCcore-10.2.0/sbin:/cm/shared/easybuild/GenuineIntel/software/util-linux/2.36-GCCcore-10.2.0/bin:/cm/shared/easybuild/software/ncurses/6.2-GCCcore-10.2.0/bin:/cm/shared/easybuild/GenuineIntel/software/freetype/2.10.3-GCCcore-10.2.0/bin:/cm/shared/easybuild/GenuineIntel/software/Brotli/1.0.9-GCCcore-10.2.0/bin:/cm/shared/easybuild/GenuineIntel/software/libpng/1.6.37-GCCcore-10.2.0/bin:/cm/shared/easybuild/GenuineIntel/software/expat/2.2.9-GCCcore-10.2.0/bin:/cm/shared/easybuild/GenuineIntel/software/bzip2/1.0.8-GCCcore-10.2.0/bin:/cm/shared/easybuild/GenuineIntel/software/FFTW/3.3.8-gompi-2020b/bin:/cm/shared/easybuild/GenuineIntel/software/OpenMPI/4.0.5-GCC-10.2.0/bin:/cm/shared/easybuild/GenuineIntel/software/PMIx/3.1.5-GCCcore-10.2.0/bin:/cm/shared/easybuild/GenuineIntel/software/libfabric/1.11.0-GCCcore-10.2.0/bin:/cm/shared/easybuild/GenuineIntel/software/UCX/1.9.0-GCCcore-10.2.0/bin:/cm/shared/easybuild/GenuineIntel/software/libevent/2.1.12-GCCcore-10.2.0/bin:/cm/shared/easybuild/GenuineIntel/software/hwloc/2.2.0-GCCcore-10.2.0/sbin:/cm/shared/easybuild/GenuineIntel/software/hwloc/2.2.0-GCCcore-10.2.0/bin:/cm/shared/easybuild/GenuineIntel/software/libxml2/2.9.10-GCCcore-10.2.0/bin:/cm/shared/easybuild/GenuineIntel/software/XZ/5.2.5-GCCcore-10.2.0/bin:/cm/shared/easybuild/GenuineIntel/software/numactl/2.0.13-GCCcore-10.2.0/bin:/cm/shared/easybuild/software/binutils/2.35-GCCcore-10.2.0/bin:/cm/shared/easybuild/software/GCCcore/10.2.0/bin:/cm/shared/apps/slurm/20.11.9/sbin:/cm/shared/apps/slurm/20.11.9/bin:/cm/local/apps/gcc/8.2.0/bin:/usr/lib/qt-3.3/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/me/.local/bin:/home/me/bin

SHARK

This is how $PATH looks like after loading R on SHARK

[rfschulz@res-hpc-lo02 ~]$ echo $PATH /share/software/statistical/R/R-4.1.2/bin:/home/me/.local/bin:/home/me/bin:/opt/TurboVNC/bin:/opt/VirtualGL/bin:/share/software/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/me/.dotnet/tools

After loading the module for R, the $PATH variable has been changed to include the directories holding executables of the newly loaded modules.

Taking this to its conclusion, module load adds software to your $PATH. It “loads” software.

Careful when you load several modules. It is possible that their dependencies can cause conflicts in particular with respect to different compilers which can create problems later on. It is best to always check what other modules have been automatically loaded.

Also a note of warning: When you load several modules, it is possible that their dependencies can cause conflicts and problems later on. It is best to always check what other modules have been automatically loaded.

Unload modules

The command module unload “un-loads” a module. Using the above example of R

ALICE

[m@nodelogin02 ~]$ module unload R/4.0.5-foss-2020b [me@nodelogin02 ~]$ module list Currently Loaded Modules: 1) shared 23) foss/2020b 45) gettext/0.21-GCCcore-10.2.0 67) HDF5/1.10.7-gompi-2020b 2) DefaultModules 24) bzip2/1.0.8-GCCcore-10.2.0 46) PCRE/8.44-GCCcore-10.2.0 68) UDUNITS/2.2.26-GCCcore-10.2.0 3) gcc/8.2.0 25) expat/2.2.9-GCCcore-10.2.0 47) GLib/2.66.1-GCCcore-10.2.0 69) GSL/2.6-GCC-10.2.0 4) slurm/20.11.9 26) libpng/1.6.37-GCCcore-10.2.0 48) cairo/1.16.0-GCCcore-10.2.0 70) Ghostscript/9.53.3-GCCcore-10.2.0 5) GCCcore/10.2.0 27) Brotli/1.0.9-GCCcore-10.2.0 49) libreadline/8.0-GCCcore-10.2.0 71) JasPer/2.0.24-GCCcore-10.2.0 6) zlib/1.2.11-GCCcore-10.2.0 28) freetype/2.10.3-GCCcore-10.2.0 50) Tcl/8.6.10-GCCcore-10.2.0 72) LittleCMS/2.11-GCCcore-10.2.0 7) binutils/2.35-GCCcore-10.2.0 29) ncurses/6.2-GCCcore-10.2.0 51) SQLite/3.33.0-GCCcore-10.2.0 73) ImageMagick/7.0.10-35-GCCcore-10.2.0 8) GCC/10.2.0 30) util-linux/2.36-GCCcore-10.2.0 52) PCRE2/10.35-GCCcore-10.2.0 74) GLPK/4.65-GCCcore-10.2.0 9) numactl/2.0.13-GCCcore-10.2.0 31) fontconfig/2.13.92-GCCcore-10.2.0 53) NASM/2.15.05-GCCcore-10.2.0 75) nodejs/12.19.0-GCCcore-10.2.0 10) XZ/5.2.5-GCCcore-10.2.0 32) xorg-macros/1.19.2-GCCcore-10.2.0 54) libjpeg-turbo/2.0.5-GCCcore-10.2.0 76) Python/3.8.6-GCCcore-10.2.0 11) libxml2/2.9.10-GCCcore-10.2.0 33) X11/20201008-GCCcore-10.2.0 55) jbigkit/2.1-GCCcore-10.2.0 77) netCDF/4.7.4-gompi-2020b 12) libpciaccess/0.16-GCCcore-10.2.0 34) gzip/1.10-GCCcore-10.2.0 56) LibTIFF/4.1.0-GCCcore-10.2.0 78) GEOS/3.9.1-GCC-10.2.0 13) hwloc/2.2.0-GCCcore-10.2.0 35) lz4/1.9.2-GCCcore-10.2.0 57) Tk/8.6.10-GCCcore-10.2.0 79) PROJ/7.2.1-GCCcore-10.2.0 14) libevent/2.1.12-GCCcore-10.2.0 36) zstd/1.4.5-GCCcore-10.2.0 58) cURL/7.72.0-GCCcore-10.2.0 80) libgeotiff/1.6.0-GCCcore-10.2.0 15) UCX/1.9.0-GCCcore-10.2.0 37) libdrm/2.4.102-GCCcore-10.2.0 59) GMP/6.2.0-GCCcore-10.2.0 81) pybind11/2.6.0-GCCcore-10.2.0 16) libfabric/1.11.0-GCCcore-10.2.0 38) libglvnd/1.3.2-GCCcore-10.2.0 60) NLopt/2.6.2-GCCcore-10.2.0 82) SciPy-bundle/2020.11-foss-2020b 17) PMIx/3.1.5-GCCcore-10.2.0 39) libunwind/1.4.0-GCCcore-10.2.0 61) libogg/1.3.4-GCCcore-10.2.0 83) libtirpc/1.3.1-GCCcore-10.2.0 18) OpenMPI/4.0.5-GCC-10.2.0 40) LLVM/11.0.0-GCCcore-10.2.0 62) FLAC/1.3.3-GCCcore-10.2.0 84) HDF/4.2.15-GCCcore-10.2.0 19) OpenBLAS/0.3.12-GCC-10.2.0 41) Mesa/20.2.1-GCCcore-10.2.0 63) libvorbis/1.3.7-GCCcore-10.2.0 85) GDAL/3.2.1-foss-2020b 20) gompi/2020b 42) libGLU/9.0.1-GCCcore-10.2.0 64) libsndfile/1.0.28-GCCcore-10.2.0 21) FFTW/3.3.8-gompi-2020b 43) pixman/0.40.0-GCCcore-10.2.0 65) ICU/67.1-GCCcore-10.2.0 22) ScaLAPACK/2.1.0-gompi-2020b 44) libffi/3.3-GCCcore-10.2.0 66) Szip/2.1.1-GCCcore-10.2.0

As you can see, the module for R was “unloaded”.

SHARK

[me@res-hpc-lo02 ~]$ module unload statistical/R/4.1.2/gcc.8.3.1 [me@res-hpc-lo02 ~]$ module list No modules loaded

IMPORTANT: Unloading a module does not unload its dependencies.

If you want to unload all currently loaded modules, you can use the command module purge.

For ALICE users

Using module purge will also remove the module for Slurm, but you can quickly load it again by running module load slurm (There is only one version of slurm available on the cluster, so no need to specify the version here)

Pro tip

You can also use module purge in your batch script before loading the modules that you need in order to make sure that no other modules can interfere with your job environment.

Saving and restoring (default) modules

If you work with a standard set of modules quite a lot, you can save and restore to the list of modules at any time.

Let’s start from a clean module environment:

module purge

ALICE

On ALICE, you should always include the module slurm. Otherwise, you will not be able to access Slurm commands when you restore the module environment

Load a set of modules and save them as default

[me@nodelogin02 ~]$ module load slurm Python/3.10.4-GCCcore-11.3.0 git/2.36.0-GCCcore-11.3.0-nodocs [me@nodelogin02 ~]$ module save Saved current collection of modules to: "default"

If you need to restore them:

[me@nodelogin02 ~]$ module restore Restoring modules from user's default

To test the effect of restoring them, either load another module or purge the module environment. You can check with module list that the default modules have been loaded.

You can can also define multiple different module environments by specifying a name when saving the module list:

[me@nodelogin02 ~]$ module load slurm Python/3.10.4-GCCcore-11.3.0 git/2.36.0-GCCcore-11.3.0-nodocs [me@nodelogin02 ~]$ module save python Saved current collection of modules to: "python"

And restoring works like this:

[me@nodelogin02 ~]$ module restore python Restoring modules from user's python

SHARK

Load a set of modules and save them as default

[me@res-hpc-lo02 ~]$ module load system/python/3.10.2 [me@res-hpc-lo02 ~]$ module save Saved current collection of modules to: "default"

If you need to restore them:

[me@res-hpc-lo02 ~]$ module restore Restoring modules from user's default

To test the effect of restoring them, either load another module or purge the module environment. You can check with module list that the default modules have been loaded.

You can can also define multiple different module environments by specifying a name when saving the module list:

[me@res-hpc-lo02 ~]$ module load system/python/3.10.2 [me@res-hpc-lo02 ~]$ module save python Saved current collection of modules to: "python"

And restoring works like this:

[me@res-hpc-lo02 ~]$ module restore python Restoring modules from user's python

Note that using module restore in your batch scripts has an impact on the reproducibility of your batch scripts. When you want to check later on what modules (and versions) were loaded for a job, this information is no longer available in your batch script.

If you want to make use of module restore, we recommend to add the command module list after the restore command in your batch script. This way, the Slurm output file for the job will contain the list of modules and versions used.

The command module save will always save the full list of modules with the version used even if you loaded the modules without specifying a version.

You can get a full list of saved module environments like this:

module savelist

Disabling an saved collection of modules can be done with

module disable <name>

where <name> should be replaced by the name of the collection. This will not entirely remove the file though. This has to be done manually.

The various collections of modules are stored in your home directory in: ~/.lmod.d

 

Related content