Too many pythons

[DusanJovic-NOAA]

We have to deal with too many python versions. We use python in several different contexts in the ufs-weather-model. The build system needs python to preprocess ccpp suite definition files. The ecflow workflow manager used by rt.sh needs the ecflow python module. Some other scripts need a github module. So what happens is we end up loading different modules at different times, or export $PATH explicitly adding location of minconda installation and it very soon becomes impossible to know which version we end up using.

Take for example the situation on Jet.

Once you login into Jet login node, you have access to system installed python:

$ which python
/usr/bin/python
$ python --version
Python 3.6.8

The default python is 3.6.0. Then the rt.sh script first loads ecflow module:

$ module load ecflow
ECF_HOST = ecflow1, ECF_PORT = 13679
$ which python
/apps/spack/linux-rocky8-x86_64/gcc-13.2.0/python-3.11.6-6gktyo4cce4mwejpb42dkucgevsaq5ja/bin/python
$ python --version
Python 3.11.6

Now the default is changed to python in 3.11.6. We then load ufs_jet.intel module file (which uses spack-stack currently):

$ module use modulefiles
$ module load ufs_jet.intel
$ which python
/mnt/lfs4/HFIP/hfv3gfs/role.epic/spack-stack/spack-stack-1.6.0/envs/unified-env-rocky8/install/intel/2021.5.0/python-3.10.13-cmoumdz/bin/python
$ python --version
Python 3.10.13

and the default python is not 3.10.13.

On Jet specifically it is suggested to install minicinda to provide github python module (why miniconda is in spack-stack directory when there is already one miniconda (and anaconda) under /contrib is separate issue):

$ export PATH=/contrib/spack-stack/miniconda3/23.11.0/envs/ufs-weather-model/bin:/contrib/spack-stack/miniconda3/23.11.0/bin:${PATH}
$ which python
/contrib/spack-stack/miniconda3/23.11.0/envs/ufs-weather-model/bin/python
$ python --version
Python 3.8.3

Now my environment has 4 pythons, 4 different versions from 5 different locations:

$ which -a python
/contrib/spack-stack/miniconda3/23.11.0/envs/ufs-weather-model/bin/python
/contrib/spack-stack/miniconda3/23.11.0/bin/python
/mnt/lfs4/HFIP/hfv3gfs/role.epic/spack-stack/spack-stack-1.6.0/envs/unified-env-rocky8/install/intel/2021.5.0/python-3.10.13-cmoumdz/bin/python
/apps/spack/linux-rocky8-x86_64/gcc-13.2.0/python-3.11.6-6gktyo4cce4mwejpb42dkucgevsaq5ja/bin/python
/usr/bin/python

Knowing at any given moment during the script execution what modules are loaded and what miniconda locations paths are exported in the PATH becomes almost impossible. Which version is the default depends on the order of loading modules and exporting PATH, the last one wins. I am not a python expert but I wonder if this is how it needs to be.

Is there any hope to clean up this mess?

[DusanJovic-NOAA]

For more discussion see #2326