Soft links to latest modules on each platform

[AlexanderRichert-NOAA]

The idea was raised within EMC to create a symlink, something like /path/to/spack-stack/latest-modules, on each platform that would point to the most recent $env/install/modulefiles/Core directory. This would be useful for developers who don't mind updating a bunch of package versions at once, and keeps them from having to update their application modulefiles, at least on develop branches. Let's try this on one or two systems to make sure it works, then it should be as simple as reassigning the symlink with each release.

spack-stack-1.8.0:

• hera:
  /contrib/spack-stack/latest-ue-intel/
  /contrib/spack-stack/latest-ue-gcc/
• jet:
  /contrib/spack-stack/latest-ue-intel/
  /contrib/spack-stack/latest-ue-gcc/
• gaea-C5:
  /ncrc/proj/epic/spack-stack/latest-ue-intel/
• gaea-C6:
  /ncrc/proj/epic/spack-stack/c6/latest-ue-intel/
• derecho:
  /glade/work/epicufsrt/contrib/spack-stack/derecho/latest-ue-intel/
  /glade/work/epicufsrt/contrib/spack-stack/derecho/latest-ue-gcc/
• hercules:
  /apps/contrib/spack-stack/latest-ue-intel/
  /apps/contrib/spack-stack/latest-ue-gcc/
• orion:
  /apps/contrib/spack-stack/latest-ue-intel/
  /apps/contrib/spack-stack/latest-ue-gcc/
• noaa-aws:
  /contrib/spack-stack-rocky8/latest-ue-intel/
• noaa-azure:
  /contrib/spack-stack-rocky8/latest-ue-intel/
• noaa-gcloud:
  /contrib/spack-stack-rocky8/latest-ue-intel/
• s4:
  /data/prod/jedi/spack-stack/latest-ue-intel/

[AlexanderRichert-NOAA]

@climbfuji @RatkoVasic-NOAA please check out the paths in the description and add symlinks on any other systems where they might get used. These are all pointing to 1.7.0 (I didn't do Gaea C6 since we didn't deploy 1.7.0 there)

module use /ncrc/proj/epic/spack-stack/latest-ue-intel && module load stack-intel && module load ..., e.g., seems to work on all of the systems in the description based on loading a few modules, which'ing a few executables, etc.

I propose we create an issue with each release to remind ourselves to update these symlinks, and try to update them on the same day across systems to avoid excessive confusion and heartache for developers.

Relatedly, I created a wiki page to document all currently supported spack-stack environments on each platform: https://github.com/JCSDA/spack-stack/wiki/Extended-list-of-release-directories If it looks good, we can coordinate dividing up the work, which I predict will be slightly tedious but shouldn't take too long.

[climbfuji]

Thanks @AlexanderRichert-NOAA . We should focus on the NOAA/EPIC platforms for the symlinks first - suggest doing S4 next.

[RatkoVasic-NOAA]

@AlexanderRichert-NOAA I created symbolic links to the latest spack-stack (1.8.0) on 10 tier1 machines.

• We have to think of some kind of reminder, task, TODO or something that will remind us on updating it with every new release.
• Also, this list should be available somewhere for people who need it.

[climbfuji]

@AlexanderRichert-NOAA I created symbolic links to the latest spack-stack (1.8.0) on 10 tier1 machines.

• We have to think of some kind of reminder, task, TODO or something that will remind us on updating it with every new release.
• Also, this list should be available somewhere for people who need it.

How about you add it to the already closed spack-stack-1.8.0 release task issue (with the list of platforms and a reference to this issue for the location)? I usually take the last releases' task issue to create the next one.

[climbfuji]

Actually, let me do this real quick.

[climbfuji]

Done. See #1278 (comment)

[RatkoVasic-NOAA]

That was quick

[climbfuji]

I see S4 is missing on this list. Is that needed?

[RatkoVasic-NOAA]

I don't have access there

[climbfuji]

That's why I am asking - JCSDA (@srherbener) or I can do that if it's required

[srherbener]

These directories might work for the S4 symlinks

/data/prod/contrib/spack-stack
/data/prod/jedi/spack-stack

I vote for /data/prod/jedi/spack-stack and if that is agreeable, I can create the links. (I don't have write permission for the /data/prod/contrib directory.

[climbfuji]

These directories might work for the S4 symlinks

/data/prod/contrib/spack-stack /data/prod/jedi/spack-stack

I vote for /data/prod/jedi/spack-stack and if that is agreeable, I can create the links. (I don't have write permission for the /data/prod/contrib directory.

Makes sense to me, thanks @srherbener. This is also the top-level spack-stack directory, and therefore matches what is done on the other NOAA platforms.

[AlexanderRichert-NOAA]

Let's close this once Steve's done with S4 and after #1335 is merged.

[srherbener]

S4 has a symlink for latest-ue-intel now. For a while, Intel is the only compiler we've been supporting on S4.

[jedipara@s4-submit ~]$ ll /data/prod/jedi/spack-stack/
total 197
drwxr-xr-x   8 jedipara domain users 11776 Aug 26  2022 ecflow-5.8.4
lrwxrwxrwx   1 jedipara domain users    66 Oct  9 20:08 latest-ue-intel -> spack-stack-1.8.0/envs/ue-intel-2021.10.0/install/modulefiles/Core
drwxr-xr-x  18 jedipara domain users 12800 Aug 26  2022 miniconda-3.9.12
drwxr-xr-x   7 jedipara domain users 11776 May 28  2023 modulefiles
drwxr-xr-x  10 jedipara domain users 12800 Feb 22  2023 mysql-8.0.31
drwxr-sr-x 375 jedipara domain users 57856 Oct  7 03:05 source-cache
drwxr-xr-x   9 jedipara domain users 12800 May 28  2023 spack-stack-1.4.0
drwxr-xr-x   9 jedipara domain users 12800 Aug 14  2023 spack-stack-1.4.1
drwxr-xr-x  10 jedipara domain users 13824 Oct 12  2023 spack-stack-1.5.0
drwxr-xr-x  10 jedipara domain users 12800 Nov 17  2023 spack-stack-1.5.1
drwxr-xr-x  10 jedipara domain users 13824 Jul  8 14:59 spack-stack-1.6.0
drwxr-xr-x  11 jedipara domain users 13824 May 23 22:40 spack-stack-1.7.0
drwxr-xr-x  11 jedipara domain users 12800 Oct  7 03:05 spack-stack-1.8.0

[climbfuji]

Thanks @srherbener! I updated the description of the issue (first comment) with the link on S4.