conda-env-mod: A tool for simplifying Python package installation on HPC clusters

Due to limited user-level permissions on managed clusters, installing and maintaining Python packages becomes a challenging task for novice users. This script uses Anaconda environments and environment modules (Lmod) to simplify this process. There are three simple steps to install and use Python packages using conda-env-mod:

1. Create an Anaconda environment using conda-env-mod. During the process, the script automatically creates module file for using the environment.
2. Load the module generated by conda-env-mod. By default modules are generated in $HOME/privatemodules. You will need to add it to your $MODULEPATH.
3. Now use conda or pip to install your Python package. Once the installation finishes, you can directly import it in your script. No need to run conda init or conda activate.

Prerequisites

• You must have Lmod module software installed and configured on your system. Currently, conda-env-mod generates module files in Lua format only. If you need to use TCL modules or would like to contribute a patch, please contact the maintainers.
• Anaconda must be installed as a module. The script assumes that $CONDA_ENVS_PATH is defined in your Anaconda module and points to a writable directory, otherwise conda create -n my_env command may fail.

Usage

User guide for conda-env-mod is provided as a manual page with this repository. You will need to add the share directory to your $MANPATH.

  $ export PATH=/path/to/conda-env-mod:$PATH
  $ export MANPATH=/path/to/conda-env-mod/share/man:$MANPATH
  $ man conda-env-mod

A shortened version of usage instructions can be obtained with --help option.

  $ conda-env-mod --help

Several use-cases of conda-env-mod is given in this webpage: Python package installation

Citation

Kindly cite usage of this tool by pointing to this repository.

Amiya K. Maji, Lev Gorenstein, and Geoffrey Lentner. "Demystifying Python Package Installation with conda-env-mod." In Proceedings of the 7th Annual Workshop on HPC User Support Tools (HUST 2020), pp. 1-10. 2020.

Maintainers

This code is maintained by Amiya K Maji ([email protected]) and Lev Gorenstein ([email protected]).

© Purdue University, 2020