spack-stack-1.8.0 release tasks #1278

climbfuji · 2024-08-29T15:19:02Z

Tasks for the spack-stack-1.8.0 release:

Create release branches
Install release candidate on a few selected systems for testing. Finalize testing/report back/fix issues by September 6. Suggestion:
- Hera (@RatkoVasic-NOAA)
- Nautilus (@climbfuji): /p/work2/heinzell/spack-stack-1.8.0-rc1/envs/* with * being one of:
  - ue-intel-2021.5.0
  - ue-gcc-12.2.1
  - ue-aocc-4.0.0 (incomplete, because many packages don't build with flang)
  - ue-oneapi-2024.1.2
  - ne-intel-2021.5.0
  - ne-gcc-12.2.1
  - ne-oneapi-2024.1.2
  - ne-aocc-4.0.0 (incomplete, because many packages don't build with flang)
- Discover SCU16 (@srherbener, @ashley314)
- Acorn (@AlexanderRichert-NOAA)
- Blackpearl (tier-2; @climbfuji)
  - [email protected] using ifx (incomplete, see Unified environment build errors with ifx #1279; complete with ifort)
Update documentation / prepare changes of documentation to move documentation specific for a specific release to the Wiki: release/1.8.0: Update site configs for Nautilus, Blackpearl, Casper; move release-specific site documentation for preconfigured sites to Wiki #1280
Roll out release starting September 9, to be completed by September 27. Update site configs and site-specific documentation until September 27.
- Acorn (EMC)
- ~~Atlantis (NRL)~~ defer to after system maintenance
- AWS-Pcluster (JCSDA, @eap)
- ~~Casper~~ (dropped to tier2)
- Derecho ~~(EPIC, @RatkoVasic-NOAA)~~ (NRL, @climbfuji)
  - ue-intel-2021.10.0
  - ue-gcc-12.2.0 (failing)
- Discover SCU16 (JCSDA, @ashley314)
  - ue-intel-2021.6.0
  - ue-gcc-12.1.0
- Discover SCU17 (JCSDA, @ashley314)
  - ue-intel-2021.10.0
  - ue-gcc-12.3.0
- Gaea C5 (EPIC) (@RatkoVasic-NOAA): /ncrc/proj/epic/spack-stack/spack-stack-1.8.0/envs/
  - ue-intel-2021.9.0 (failing)
- Gaea C6 (EPIC) (@RatkoVasic-NOAA): /ncrc/proj/epic/spack-stack/c6/spack-stack-1.8.0/envs/
  - ue-intel-2021.10.0 (failing)
- Hera (@RatkoVasic-NOAA): /contrib/spack-stack/spack-stack-1.8.0/envs/
  - ue-intel-2021.5.0
  - ue-gcc-9.2.0
- Hercules (EPIC) (@RatkoVasic-NOAA): /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.8.0/envs/
  - ue-intel-2021.9.0
  - ue-gcc-12.2.0
- Jet (EPIC) (@RatkoVasic-NOAA): /contrib/spack-stack/spack-stack-1.8.0/envs/
  - ue-intel-2021.5.0
  - ue-gcc-9.2.0
- Narwhal (NRL)
  - ne-intel-2021.10.0
  - ue-intel-2021.10.0 (chained from ne-intel-2021.10.0)
  - ne-gcc-10.3.0
  - ue-gcc-10.3.0 (chained from ne-gcc-10.3.0)
  - readthedocs updated
  - wiki updated
  - chmod
- Nautilus (NRL)
  - ne-intel-2021.5.0
  - ue-intel-2021.5.0 (chained from ne-intel-2021.5.0)
  - ne-oneapi-2024.2.1
  - ue-oneapi-2024.2.1 (chained from ne-oneapi-2024.2.1)
  - ne-gcc-12.2.1
  - ue-gcc-12.2.1 (chained from ne-gcc-12.2.1)
  - readthedocs updated
  - wiki updated
  - chmod
- Parallelworks EPIC AWS (@RatkoVasic-NOAA): /contrib/spack-stack-rocky8/spack-stack-1.8.0/envs/
  - ue-intel-2021.10.0
- Parallelworks EPIC Azure (@RatkoVasic-NOAA) /contrib/spack-stack-rocky8/spack-stack-1.8.0/envs/
  - ue-intel-2021.10.0
- Parallelworks EPIC GCP (@RatkoVasic-NOAA) /contrib/spack-stack-rocky8/spack-stack-1.8.0/envs/
  - ue-intel-2021.10.0
- ~~Parallelworks JCSDA GCP (@fmahebert)~~ no longer supported
- Orion (EPIC) (@RatkoVasic-NOAA): /work/noaa/epic/role-epic/spack-stack/orion/spack-stack-1.8.0/envs/
  - ue-intel-2021.9.0
  - ue-gcc-12.2.0
- S4 ~~(JCSDA, @srherbener)~~ (NRL, @climbfuji)
- ~~Sandy (NRL)~~ defer to after system maintenance
Create soft links to latest modules on each NOAA RDHPCS platform
- See Soft links to latest modules on each platform #1204 for a list of platforms and locations
Draft release notes
Tag spack submodule, update .gitmodules and spack submodule pointer in spack-stack, tag spack-stack
Publish release on ~~Friday, September 27~~ Wednesday, October 2, 2024
Merge release branches back into develop / spack-stack-dev

The text was updated successfully, but these errors were encountered:

RatkoVasic-NOAA · 2024-08-29T18:02:51Z

Started working on Hera. It concretized OK.
Installation in under way. There is one problem with crtm_fix:

==> Installing crtm-fix-2.4.0.1_emc-al2juhzkilqssvyg75jsni664uhcknls [21/291]
==> No binary for crtm-fix-2.4.0.1_emc-al2juhzkilqssvyg75jsni664uhcknls found: installing from source
==> Error: FetchError: All fetchers failed for spack-stage-crtm-fix-2.4.0.1_emc-al2juhzkilqssvyg75jsni664uhcknls
==> Warning: Skipping build of crtm-2.4.0.1-fvusealmjzqzvr2tjoe7b4gfxkhfil4x since crtm-fix-2.4.0.1_emc-al2juhzkilqssvyg75jsni664uhcknls failed

I saw this on Jet as well. It looks like we cannot access crtm-fix-2.4.0.1_emc.

and another:

==> Installing libpthread-stubs-0.5-75nwhcat2k4nkuzlmnzow5iisqi5x4ct [35/291]
==> No binary for libpthread-stubs-0.5-75nwhcat2k4nkuzlmnzow5iisqi5x4ct found: installing from source
==> Error: FetchError: All fetchers failed for spack-stage-libpthread-stubs-0.5-75nwhcat2k4nkuzlmnzow5iisqi5x4ct
==> Warning: Skipping build of libxcb-1.16-bdkenjctf4jxkkzk73cnoiztl6cqjztm since libpthread-stubs-0.5-75nwhcat2k4nkuzlmnzow5iisqi5x4ct failed

climbfuji · 2024-08-29T19:35:47Z

Can you try to rerun the install for crtm-fix with spack -d install ... - that might give you more information. If you can't access those files from hera (which shouldn't be the case), then you can try to create a mirror on another machine and update the spack source-cache on hera.

RatkoVasic-NOAA · 2024-08-29T19:40:06Z

Can you try to rerun the install for crtm-fix with spack -d install ... - that might give you more information. If you can't access those files from hera (which shouldn't be the case), then you can try to create a mirror on another machine and update the spack source-cache on hera.

Didn't try for crtm-fix, but there are others that failed with "All fetchers failed", but when I tried to download them manually on Hera, download was OK.

srherbener · 2024-08-29T20:13:58Z

@climbfuji I'm going to start the build on Discover SCU16. Where do I place the release candidate, and what do I name it? Perhaps: /gpfsm/dswdev/jcsda/spack-stack/scu16/spack-stack-rc-1.8.0

climbfuji · 2024-08-29T20:25:53Z

I use spack-stack-1.8.0-rc1 but it's up to you of course. If the path is longer than the final spack-stack-1.8.0 install, then you can use the release candidate to create a binary cache and install from it when you do the final 1.8.0 install.

srherbener · 2024-08-29T21:33:17Z

Thanks @climbfuji. I just wanted to make sure I'm following a best practice. Not sure I understand your comment about the binary cache. I created /gpfsm/dswdev/jcsda/spack-stack/scu16/spack-stack-rc-1.8.0 and will that satisfy the requirement to be able to use the binary cache for the final install?

climbfuji · 2024-08-29T21:40:53Z

If you use the same names for the environments, then yes. spack-stack-1.8.0 has three characters less than spack-stack-rc-1.8.0.

RatkoVasic-NOAA · 2024-08-30T12:01:40Z

Hera:
installed ue-intel (had to use mirror from Hercules): /contrib/spack-stack/spack-stack-1.8.0/envs/ue-intel
ue-gcc installation in progress: /contrib/spack-stack/spack-stack-1.8.0/envs/ue-gcc
For now, only change goes to site/mirrors.yaml

climbfuji · 2024-08-30T14:40:53Z

@RatkoVasic-NOAA I am confused a bit - because of the recent changes to how we build ecflow, I was expecting site config and documentation updates for Hera - removal of external ecflow, addition of external qt5 if not already there, etc. See #1257

srherbener · 2024-08-30T16:59:15Z

I am building the release candidates on Discover SCU16, and @ashley314 will test these.

GCC: /gpfsm/dswdev/jcsda/spack-stack/scu16/spack-stack-rc-1.8.0/envs/ue-gcc-12.1.0
Intel: /gpfsm/dswdev/jcsda/spack-stack/scu16/spack-stack-rc-1.8.0/envs/ue-intel-2021.6.0

Checked box means the environment is ready to test.

climbfuji · 2024-08-30T17:00:33Z

I am building the release candidates on Discover SCU16, and @ashley314 will test these.

GCC: /gpfsm/dswdev/jcsda/spack-stack/scu16/spack-stack-rc-1.8.0/envs/ue-gcc-12.1.0

Intel: /gpfsm/dswdev/jcsda/spack-stack/scu16/spack-stack-rc-1.8.0/envs/ue-intel-2021.6.0

Checked box means the environment is ready to test.

Just want to double-check that you made the ecflow/qt5 changes (see #1257)?

srherbener · 2024-08-30T17:10:51Z

Sorry I didn't add ecflow to the config - too many things going on at once. I'll go do that now.

climbfuji · 2024-08-30T17:12:05Z

We don't want ecflow in the site configs. We want only qt@5 so that spack-stack builds ecflow. (Basically what we did on S4)

climbfuji · 2024-08-30T17:13:13Z

Therefore, we can likely remove the external miniconda and ecflow dependencies for spack-stack, too.

climbfuji · 2024-08-30T17:14:19Z

Therefore, we can likely remove the external miniconda and ecflow dependencies for spack-stack, too.

Whether you then need a module use /some/path/to/jedis/modules at all or not depends on where the qt and mysql are coming from.

srherbener · 2024-08-30T19:28:46Z

@ashley314 Discover SCU16 gcc environment is now ready for testing. It includes ecflow in the stack (and qt@5 is an external package).

srherbener · 2024-08-30T23:25:32Z

@ashley314 Discover SCU16 gcc environment is now ready for testing. It includes ecflow in the stack (and qt@5 is an external package).

Discover SCU16 Intel environment is now ready for testing.

DavidHuber-NOAA · 2024-09-16T15:53:13Z

Could the gsi-addon environment also be installed on these platforms: Hera, Jet, Orion, Hercules, Gaea C5 (C6 optional), all three PW CSPs, and S4. EDIT: This environment is only required for the Intel compiler suite.

RatkoVasic-NOAA · 2024-09-16T15:56:46Z

@DavidHuber-NOAA Yes, as soon as we install 1.8.0 on those machines. Some of them are ready for addons (Hera and Jet).

RatkoVasic-NOAA · 2024-09-16T23:14:15Z

@DavidHuber-NOAA I added GSI addon to Hera spack-stack-1.8.0:
/contrib/spack-stack/spack-stack-1.8.0/envs/gsi-intel-2021.5.0/install/modulefiles/Core
Notice some name changes.
Also, here are changes for GSI modulefiles:

gsi_hera.intel.lua:
prepend_path("MODULEPATH", "/contrib/spack-stack/spack-stack-1.8.0/envs/gsi-intel-2021.5.0/install/modulefiles/Core")
local python_ver=os.getenv("python_ver") or "3.11.7"
local stack_impi_ver=os.getenv("stack-intel-oneapi-mpi") or "2021.5.1"
local cmake_ver=os.getenv("cmake_ver") or "3.28.1"

gsi_common.lua:
local bufr_ver=os.getenv("bufr_ver") or "12.1.0"
local ip_ver=os.getenv("ip_ver") or "5.0.0"

climbfuji · 2024-09-16T23:28:58Z

Please make sure all site config and documentation updates are in the spack-stack release branch (incl the readthedocs documentation if applicable) and in https://github.com/JCSDA/spack-stack/wiki/spack%E2%80%90stack%E2%80%901.8.0-release-documentation - THANK YOU!

RatkoVasic-NOAA · 2024-09-16T23:50:52Z

@DavidHuber-NOAA Just finished GSI addon on Jet:
/contrib/spack-stack/spack-stack-1.8.0/envs/gsi-intel-2021.5.0/install/modulefiles/Core
Exactly same changes to Jet modulefiles like for Hera.

DavidHuber-NOAA · 2024-09-17T12:29:30Z

Great, thank you @RatkoVasic-NOAA!

ashley314 · 2024-09-19T16:45:51Z

FYI, I found a typo for discover-scu17 in configs/sites/tier1/discover-scu17/packages_intel.yaml.

-    - spec: [email protected]%[email protected]
+    - spec: [email protected]%[email protected]

climbfuji · 2024-09-19T16:58:50Z

FYI, I found a typo for discover-scu17 in configs/sites/tier1/discover-scu17/packages_intel.yaml.
-    - spec: [email protected]%[email protected]
+    - spec: [email protected]%[email protected]

Please create a PR for release/1.8.0 with all changes needed for the discover platforms. Thanks!

RatkoVasic-NOAA · 2024-09-25T01:59:46Z

@DavidHuber-NOAA Current list of gsi addons to 1.8.0:

Hera: /contrib/spack-stack/spack-stack-1.8.0/envs/gsi-intel-2021.5.0/install/modulefiles/Core
Jet: /contrib/spack-stack/spack-stack-1.8.0/envs/gsi-intel-2021.5.0/install/modulefiles/Core
Gaea-C5: /ncrc/proj/epic/spack-stack/c6/spack-stack-1.8.0/envs/gsi-intel-2021.10.0/install/modulefiles/Core
Gaea-C6: /ncrc/proj/epic/spack-stack/spack-stack-1.8.0/envs/gsi-intel-2021.9.0/install/modulefiles/Core
NOAA-AWS: /contrib/spack-stack-rocky8/spack-stack-1.8.0/envs/gsi-intel-2021.10.0/install/modulefiles/Core

climbfuji added this to spack-stack-1.8.0 (2024 Q3) Aug 29, 2024

climbfuji converted this from a draft issue Aug 29, 2024

climbfuji assigned climbfuji, AlexanderRichert-NOAA, natalie-perlin, RatkoVasic-NOAA, srherbener and eap Aug 29, 2024

climbfuji moved this from Todo to In Progress in spack-stack-1.8.0 (2024 Q3) Aug 29, 2024

climbfuji mentioned this issue Aug 30, 2024

release/1.8.0: Update site configs for Nautilus, Blackpearl, Casper; move release-specific site documentation for preconfigured sites to Wiki #1280

Merged

3 tasks

srherbener mentioned this issue Aug 30, 2024

spack-stack-1.8.0 config for Discover #1282

Merged

3 tasks

ashley314 assigned ashley314 and unassigned RatkoVasic-NOAA Sep 3, 2024

ashley314 assigned RatkoVasic-NOAA Sep 3, 2024

AlexanderRichert-NOAA mentioned this issue Sep 4, 2024

Acorn updates for 1.8.0 #1285

Merged

3 tasks

climbfuji mentioned this issue Sep 12, 2024

release/1.8.0: next round of NRL site config/doc updates #1298

Merged

3 tasks

AlexanderRichert-NOAA mentioned this issue Sep 17, 2024

Update Acorn doco for 1.8.0 #1301

Merged

3 tasks

ashley314 mentioned this issue Sep 20, 2024

update for discover-scu17 mkl and docs #1306

Merged

3 tasks

RatkoVasic-NOAA mentioned this issue Sep 25, 2024

Update changes from tier1 machines for 1.8.0 release #1310

Merged

3 tasks

This was referenced Sep 30, 2024

release/1.8.0: Update derecho site config #1315

Merged

Merge release/1.8.0 into develop #1321

Merged

RatkoVasic-NOAA mentioned this issue Oct 3, 2024

Update site/packages.yaml and site/modules.yaml #1322

Merged

3 tasks

climbfuji mentioned this issue Oct 4, 2024

Cherry-pick 87067e9 (Update site/packages.yaml and site/modules.yaml (#1322)) from release/1.8.0 #1323

Merged

3 tasks

RatkoVasic-NOAA mentioned this issue Oct 5, 2024

fix openmpi version on Hercules #1326

Merged

3 tasks

climbfuji mentioned this issue Oct 7, 2024

For release/1.8.0: update S4 site config and documentation #1328

Merged

3 tasks

RatkoVasic-NOAA mentioned this issue Oct 8, 2024

Release 1.8.0 additional fixes for Jet, Hercules and Orion (after running ufs-weather-model) #1331

Merged

3 tasks

climbfuji mentioned this issue Oct 8, 2024

Cherry-pick last three commits from release/1.8.0 for develop #1332

Merged

3 tasks

climbfuji closed this as completed in #1332 Oct 8, 2024

github-project-automation bot moved this from In Progress to Done in spack-stack-1.8.0 (2024 Q3) Oct 8, 2024

climbfuji mentioned this issue Oct 9, 2024

Soft links to latest modules on each platform #1204

Closed

11 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

spack-stack-1.8.0 release tasks #1278

spack-stack-1.8.0 release tasks #1278

climbfuji commented Aug 29, 2024 •

edited

Loading

RatkoVasic-NOAA commented Aug 29, 2024 •

edited

Loading

climbfuji commented Aug 29, 2024

RatkoVasic-NOAA commented Aug 29, 2024

srherbener commented Aug 29, 2024

climbfuji commented Aug 29, 2024

srherbener commented Aug 29, 2024

climbfuji commented Aug 29, 2024

RatkoVasic-NOAA commented Aug 30, 2024 •

edited

Loading

climbfuji commented Aug 30, 2024

srherbener commented Aug 30, 2024 •

edited

Loading

climbfuji commented Aug 30, 2024 •

edited by srherbener

Loading

srherbener commented Aug 30, 2024

climbfuji commented Aug 30, 2024 •

edited

Loading

climbfuji commented Aug 30, 2024

climbfuji commented Aug 30, 2024

srherbener commented Aug 30, 2024

srherbener commented Aug 30, 2024

DavidHuber-NOAA commented Sep 16, 2024 •

edited

Loading

RatkoVasic-NOAA commented Sep 16, 2024

RatkoVasic-NOAA commented Sep 16, 2024

climbfuji commented Sep 16, 2024

RatkoVasic-NOAA commented Sep 16, 2024

DavidHuber-NOAA commented Sep 17, 2024

ashley314 commented Sep 19, 2024

climbfuji commented Sep 19, 2024

RatkoVasic-NOAA commented Sep 25, 2024 •

edited

Loading

spack-stack-1.8.0 release tasks #1278

spack-stack-1.8.0 release tasks #1278

Comments

climbfuji commented Aug 29, 2024 • edited Loading

RatkoVasic-NOAA commented Aug 29, 2024 • edited Loading

climbfuji commented Aug 29, 2024

RatkoVasic-NOAA commented Aug 29, 2024

srherbener commented Aug 29, 2024

climbfuji commented Aug 29, 2024

srherbener commented Aug 29, 2024

climbfuji commented Aug 29, 2024

RatkoVasic-NOAA commented Aug 30, 2024 • edited Loading

climbfuji commented Aug 30, 2024

srherbener commented Aug 30, 2024 • edited Loading

climbfuji commented Aug 30, 2024 • edited by srherbener Loading

srherbener commented Aug 30, 2024

climbfuji commented Aug 30, 2024 • edited Loading

climbfuji commented Aug 30, 2024

climbfuji commented Aug 30, 2024

srherbener commented Aug 30, 2024

srherbener commented Aug 30, 2024

DavidHuber-NOAA commented Sep 16, 2024 • edited Loading

RatkoVasic-NOAA commented Sep 16, 2024

RatkoVasic-NOAA commented Sep 16, 2024

climbfuji commented Sep 16, 2024

RatkoVasic-NOAA commented Sep 16, 2024

DavidHuber-NOAA commented Sep 17, 2024

ashley314 commented Sep 19, 2024

climbfuji commented Sep 19, 2024

RatkoVasic-NOAA commented Sep 25, 2024 • edited Loading

climbfuji commented Aug 29, 2024 •

edited

Loading

RatkoVasic-NOAA commented Aug 29, 2024 •

edited

Loading

RatkoVasic-NOAA commented Aug 30, 2024 •

edited

Loading

srherbener commented Aug 30, 2024 •

edited

Loading

climbfuji commented Aug 30, 2024 •

edited by srherbener

Loading

climbfuji commented Aug 30, 2024 •

edited

Loading

DavidHuber-NOAA commented Sep 16, 2024 •

edited

Loading

RatkoVasic-NOAA commented Sep 25, 2024 •

edited

Loading