Add AWS functionality to workflow setup and forecast job #1667

HenryRWinterbottom · 2023-06-06T21:20:40Z

Description

This PR provides support for running UFS weather model forecasts, using the global-workflow, on the AWS Parallel Works (PW) platform. This PR accomplishes the following:

Building the UFS weather model within the global-workflow build system;
Running a UFS weather model atmosphere-only deterministic forecast using the global-workflow infrastructure.

Follow-up PRs will include the following:

Adding documentation for building and using the AWS PW UFSWM spack-stack environment; this is heavily borrowed from Sadegh Sadeghi Tabas's presentation 03 May 2023;
Addressing remaining TODO issues within this PR; several notes are made throughout as to how to better streamline the connections between the UFSWM and the global-workflow for the respective cloud platforms;
Adding data-transfer capabilities within the global-workflow for cloud deployments;
Testing additional UFSWM capabilities (e.g., coupling, etc.,) and resolutions (this PR has been tested at C48 and has not been configured for other cubed-sphere resolutions).

No dependencies other than the loading of the appropriate AWS PW modules are required. These are included in this PR.

Type of change

Please delete options that are not relevant.

New feature (non-breaking change which adds functionality)
This change requires a documentation update

How Has This Been Tested?

This PR has been tested as follows:

The global-workflow fork/branch has been deployed to the AWS PW platform;
The UFSWM is built using the global-workflow build system;
A C48 atmosphere-only forecast has been run to completion.

Checklist

My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
My changes need updates to the documentation. I have made corresponding changes to the documentation
My changes generate no new warnings
New and existing tests pass with my changes
Any dependent changes have been merged and published

…nryWinterbottom-NOAA/global-workflow into feature/cloud_aws_fcstonly

github-advanced-security

shellcheck found more than 10 potential problems in the proposed changes. Check the Files changed tab for more details.

Fixed hell-norm detected bug.

jobs/rocoto/fcst.sh

Updates for UFSWM develop branch.

sorc/build_ufs.sh

aerorahul

Sorry for getting late on this PR.
I am noticing redefinitions between the global-workflow and the ufs-weather-model e.g. PW_CSP and some of the module loading.
Please take a look at the comments.

aerorahul · 2023-06-09T19:38:58Z

ush/detect_machine.sh

@@ -38,6 +38,16 @@ case $(hostname -f) in
  *) MACHINE_ID=UNKNOWN ;;  # Unknown platform
 esac

+if [[ ${MACHINE_ID} == "UNKNOWN" ]]; then 
+   case $(dnsdomainname -f) in


Suggested change

case $(dnsdomainname -f) in

case ${PW_CSP:-} in

Addressed in latest commit/update.

aerorahul · 2023-06-09T19:40:42Z

ush/detect_machine.sh

+   case $(dnsdomainname -f) in
+
+      # AWS Parallel Works
+      *pw-noaa*pw.local) MACHINE_ID=aws ;; 


Suggested change

*pw-noaa*pw.local) MACHINE_ID=aws ;;

"aws" | "google" | "azure" ) MACHINE_ID=noaacloud ;;

Addressed in latest commit/update.

ush/module-setup.sh

aerorahul · 2023-06-09T19:42:28Z

workflow/hosts.py

@@ -15,14 +15,15 @@ class Host:
    """

    SUPPORTED_HOSTS = ['HERA', 'ORION', 'JET',
-                       'WCOSS2', 'S4', 'CONTAINER']
+                       'WCOSS2', 'S4', 'CONTAINER', 'AWSPW']


Suggested change

'WCOSS2', 'S4', 'CONTAINER', 'AWSPW']

'WCOSS2', 'S4', 'CONTAINER', 'NOAACLOUD']

May be there is a need to distinguish between Azure, Google and AWS later.
But for now, lets keep with the ufs-weather-model

aerorahul · 2023-06-09T19:45:27Z

sorc/build_ufs.sh

+if [[ "${MACHINE_ID}" == "noaacloud" ]]; then
+  case $(dnsdomainname -f) in   
+     # TODO: Add Google and Azure platforms.
+      *pw-noaa*pw.local) CLOUD_MACHINE_ID="aws.${RT_COMPILER}" ;;
+      *) ;; 
+  esac


we shouldn't be doing detection again. MACHINE ID has been detected once. UFSWM is using a PW_CSP variable. Lets use that and if necessary use PW_CSP to distinguish.

@aerorahul Can you point me to a use case within the UFSWM of the PW_CSP instance?

For clarification, the above detection was needed to determine which CSP the user is on. The DNS strings should be unique. This allows the GW (and build system) to determine which host they are on and thus make the downstream decisions accordingly.

For example, AWS PW use mpiexec as it's parallel executable launcher. However, Azure or Google could use another. This may ultimately wind up being redundant but since I haven't see the PW_CSP use case I felt it unsafe to make such an assumption.

See https://github.com/ufs-community/ufs-weather-model/blob/develop/tests/detect_machine.sh
I would recommend a PW_CSP as a separate configurable if you wish to identify a CSP available on NOAACloud.

aerorahul · 2023-06-09T19:46:09Z

sorc/build_ufs.sh

+  if [[ "${CLOUD_MACHINE_ID}" == "aws.intel" ]]; then
+    module use /contrib/spack-stack/envs/ufswm/install/modulefiles/Core
+    module load stack-intel
+    module load stack-intel-oneapi-mpi
+    module load ufs-weather-model-env/1.0.0
+    # TODO: It is still uncertain why this is the only module that is 
+    # missing; check the spack build as this needed to be added manually.
+    module load w3emc/2.9.2 # TODO: This has similar issues for the EPIC stack.
+    module list
+  fi


I don't think this is necessary as ufs_noaacloud.intel.lua contains the information. Can we not use that?

The path that the module file points to, an EPIC stack, does not exist. As a result, the build environment cannot be defined using the module that you noted above.

[Henry.Winterbottom@hwufscpldcld-2 modulefiles]$ ls /contrib/EPIC/spack-stack/spack-stack-1.3.0/ ls: cannot access /contrib/EPIC/spack-stack/spack-stack-1.3.0/: No such file or directory

Note that this comes from the develop branch.

I suggest that we include this bit for now. Once EPIC provides a working stack we can create a new PR and make the update.

Understood. I did not know that the EPIC stack is not available for use by us. This certainly makes porting an issue.
I will discuss with @arunchawla-NOAA on how to enable that.

ush/detect_machine.sh

sorc/build_ufs.sh

aerorahul · 2023-06-13T14:23:10Z

@SadeghTabas-NOAA Would you please review the changes in the build and run script requirements of the ufs-weather-model?

SadeghTabas-NOAA · 2023-06-13T15:08:32Z

@aerorahul Sure, I'll do it today. Thanks

aerorahul

I only went through the shell scripts here.
Take a look if you agree with these suggestions.

jobs/rocoto/fcst.sh

sorc/build_ufs.sh

ush/detect_machine.sh

Co-authored-by: Rahul Mahajan <[email protected]>

WalterKolczynski-NOAA

No additional requests beyond what Rahul has pointed out. I would have rather kept $MACHINE as the specific cloud platform, but understand the decision to follow what UFS has done.

Co-authored-by: Rahul Mahajan <[email protected]>

jobs/rocoto/fcst.sh

workflow/hosts.py

Co-authored-by: Rahul Mahajan <[email protected]>

aerorahul

looks good.
You will have to show us how to run this beast on the cloud.

…)" This reverts commit 38cd0bc.

…)" (#1707) This reverts commit 38cd0bc.

HenryRWinterbottom and others added 10 commits June 6, 2023 11:03

Added cloud machine identification pieces.

dcac749

Updates for Rocoto.

27d63bc

Cloud platform modules.

61e4915

Cloud platform workflow information.

3fd30dd

Cloud platform workflow information.

3456297

UFS build script.

4bda331

Merge branch 'feature/cloud_aws_fcstonly' of http://www.github.com/He…

64de5ed

…nryWinterbottom-NOAA/global-workflow into feature/cloud_aws_fcstonly

Build system updates.

5e09606

Bug fix.

0aebce6

Updates for GW atmosphere-only C48 forecast.

a2cd9e7

HenryRWinterbottom requested review from WalterKolczynski-NOAA and aerorahul June 6, 2023 21:20

HenryRWinterbottom self-assigned this Jun 6, 2023

github-advanced-security bot found potential problems Jun 6, 2023

View reviewed changes

HenryRWinterbottom and others added 7 commits June 6, 2023 16:50

Removed left over debug statement.

9a1ff2c

Update build_ufs.sh

2a23028

Fixed hell-norm detected bug.

Update detect_machine.sh

7cc09bf

Update detect_machine.sh

47ecb86

Update build_ufs.sh

798144d

Update fcst.sh

cbd65ee

Shell norm fixes.

c900c8f

github-advanced-security bot found potential problems Jun 7, 2023

View reviewed changes

jobs/rocoto/fcst.sh Fixed Show fixed Hide fixed

HenryRWinterbottom and others added 3 commits June 7, 2023 12:36

Fixing another shellnorm detected error.

8b1bfc7

Updates for UFSWM develop branch.

0a1a4db

Merge pull request #3 from HenryWinterbottom-NOAA/test/ufswm_develop

01a16b4

Updates for UFSWM develop branch.

github-advanced-security bot found potential problems Jun 8, 2023

View reviewed changes

sorc/build_ufs.sh Fixed Show fixed Hide fixed

sorc/build_ufs.sh Fixed Show fixed Hide fixed

sorc/build_ufs.sh Fixed Show fixed Hide fixed

sorc/build_ufs.sh Fixed Show fixed Hide fixed

sorc/build_ufs.sh Fixed Show fixed Hide fixed

sorc/build_ufs.sh Fixed Show fixed Hide fixed

HenryRWinterbottom added 3 commits June 8, 2023 15:52

Shellnorm fixes.

f15db2d

Removed no longer necessary files.

e3ce0f5

Shellnorm fixes.

c07d5d5

aerorahul requested changes Jun 9, 2023

View reviewed changes

HenryRWinterbottom added 2 commits June 12, 2023 18:49

Using PW_CSP to detect platform.

9891b91

Cleanup.

49b1980

github-advanced-security bot found potential problems Jun 12, 2023

View reviewed changes

ush/detect_machine.sh Fixed Show fixed Hide fixed

sorc/build_ufs.sh Fixed Show fixed Hide fixed

Shellnorm fixes.

0a4a508

WalterKolczynski-NOAA changed the title ~~Feature/cloud aws fcstonly~~ Add AWS functionality to build workflow and run forecast job Jun 13, 2023

WalterKolczynski-NOAA changed the title ~~Add AWS functionality to build workflow and run forecast job~~ Add AWS functionality to workflow setup and forecast job Jun 13, 2023

Merge branch 'develop' into feature/cloud_aws_fcstonly

b30b5af

aerorahul requested changes Jun 16, 2023

View reviewed changes

HenryRWinterbottom and others added 2 commits June 16, 2023 09:01

Update jobs/rocoto/fcst.sh

5010175

Co-authored-by: Rahul Mahajan <[email protected]>

Update jobs/rocoto/fcst.sh

0b973f4

Co-authored-by: Rahul Mahajan <[email protected]>

WalterKolczynski-NOAA reviewed Jun 16, 2023

View reviewed changes

HenryRWinterbottom and others added 6 commits June 16, 2023 12:12

Update sorc/build_ufs.sh

2977f7e

Co-authored-by: Rahul Mahajan <[email protected]>

Update sorc/build_ufs.sh

a0ebc89

Co-authored-by: Rahul Mahajan <[email protected]>

Update ush/detect_machine.sh

0838670

Co-authored-by: Rahul Mahajan <[email protected]>

Update sorc/build_ufs.sh

235463f

Co-authored-by: Rahul Mahajan <[email protected]>

Update sorc/build_ufs.sh

0889d7e

Co-authored-by: Rahul Mahajan <[email protected]>

Update ush/module-setup.sh

67ec7b1

Co-authored-by: Rahul Mahajan <[email protected]>

aerorahul reviewed Jun 16, 2023

View reviewed changes

jobs/rocoto/fcst.sh Outdated Show resolved Hide resolved

aerorahul added 2 commits June 16, 2023 14:26

Update jobs/rocoto/fcst.sh

a7c56df

Apply suggestions from code review

bb25533

aerorahul reviewed Jun 16, 2023

View reviewed changes

workflow/hosts.py Outdated Show resolved Hide resolved

Update workflow/hosts.py

68a3c47

Co-authored-by: Rahul Mahajan <[email protected]>

aerorahul approved these changes Jun 16, 2023

View reviewed changes

WalterKolczynski-NOAA approved these changes Jun 16, 2023

View reviewed changes

aerorahul merged commit 38cd0bc into NOAA-EMC:develop Jun 18, 2023

WalterKolczynski-NOAA added a commit that referenced this pull request Jun 22, 2023

Revert "Add AWS functionality to workflow setup and forecast job (#1667…

9ac9a3c

…)" This reverts commit 38cd0bc.

WalterKolczynski-NOAA mentioned this pull request Jun 22, 2023

Revert "Add AWS functionality to workflow setup and forecast job" #1707

Merged

WalterKolczynski-NOAA added a commit that referenced this pull request Jun 22, 2023

Revert "Add AWS functionality to workflow setup and forecast job (#1667…

cc1bea7

…)" (#1707) This reverts commit 38cd0bc.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add AWS functionality to workflow setup and forecast job #1667

Add AWS functionality to workflow setup and forecast job #1667

HenryRWinterbottom commented Jun 6, 2023

github-advanced-security bot left a comment

aerorahul left a comment

aerorahul Jun 9, 2023

HenryRWinterbottom Jun 12, 2023

aerorahul Jun 9, 2023

HenryRWinterbottom Jun 12, 2023

aerorahul Jun 9, 2023

aerorahul Jun 9, 2023

aerorahul Jun 9, 2023

HenryRWinterbottom Jun 12, 2023

aerorahul Jun 12, 2023

aerorahul Jun 9, 2023

HenryRWinterbottom Jun 12, 2023

aerorahul Jun 12, 2023

aerorahul commented Jun 13, 2023

SadeghTabas-NOAA commented Jun 13, 2023

aerorahul left a comment

WalterKolczynski-NOAA left a comment

aerorahul left a comment

	pw-noaapw.local) MACHINE_ID=aws ;;
	"aws" \| "google" \| "azure" ) MACHINE_ID=noaacloud ;;

	'WCOSS2', 'S4', 'CONTAINER', 'AWSPW']
	'WCOSS2', 'S4', 'CONTAINER', 'NOAACLOUD']

Add AWS functionality to workflow setup and forecast job #1667

Add AWS functionality to workflow setup and forecast job #1667

Conversation

HenryRWinterbottom commented Jun 6, 2023

github-advanced-security bot left a comment

Choose a reason for hiding this comment

aerorahul left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aerorahul commented Jun 13, 2023

SadeghTabas-NOAA commented Jun 13, 2023

aerorahul left a comment

Choose a reason for hiding this comment

WalterKolczynski-NOAA left a comment

Choose a reason for hiding this comment

aerorahul left a comment

Choose a reason for hiding this comment