Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jenkins Pipeline Updates #2815

Conversation

TerrenceMcGuinness-NOAA
Copy link
Collaborator

@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA commented Aug 7, 2024

Description

Pipeline updates:

  • get gh location on remote machine and define global bash env variable $GH for GitHub CLI
  • Failed cases are now displayed accordingly in the Jenkins dashboard (see NOTE below)
  • Added the Build # in messaging for clarity when running from re-runs.
  • Replaced Matrix construct for concurrency with the parallel method that can use dynamic case lists
    • With removing of the hard coded list of cases we now get list of cases dynamically from the PR case directory
    • See new look of dashboard below (has more annotations and displays only used cases)

NOTE: failFast (quitting all cases on failing of one) still does not work because it isn't quitting the running remote shells. We can make this a configurable capability in feature request with some custom code. The current behavior has the remaining cases continuing to run after a FAIL label has been issued and it is incumbent of the code manager to kill the CI job in the controller before resetting another Ready label.

Type of change

  • Maintenance (code refactor, clean-up, new CI test, etc.)

How has this been tested?

Ran in the authoritative repo with full self-tests

updateddashboard

@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA added the CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules label Aug 7, 2024
ci/Jenkinsfile Outdated Show resolved Hide resolved
@emcbot emcbot added CI-Hercules-Building **Bot use only** CI testing is cloning/building on Hercules and removed CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules labels Aug 7, 2024
ci/Jenkinsfile Outdated Show resolved Hide resolved
@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA removed the CI-Hercules-Building **Bot use only** CI testing is cloning/building on Hercules label Aug 7, 2024
@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA added the CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules label Aug 7, 2024
@emcbot emcbot added CI-Hercules-Running **Bot use only** CI testing on Hercules for this PR is in-progress CI-Hercules-Failed **Bot use only** CI testing on Hercules for this PR has failed and removed CI-Hercules-Running **Bot use only** CI testing on Hercules for this PR is in-progress CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules CI-Hercules-Failed **Bot use only** CI testing on Hercules for this PR has failed labels Aug 7, 2024
@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA added CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules and removed CI-Hercules-Failed **Bot use only** CI testing on Hercules for this PR has failed labels Aug 7, 2024
@emcbot emcbot added CI-Hercules-Building **Bot use only** CI testing is cloning/building on Hercules and removed CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules labels Aug 7, 2024
@emcbot emcbot added CI-Hercules-Running **Bot use only** CI testing on Hercules for this PR is in-progress and removed CI-Hercules-Building **Bot use only** CI testing is cloning/building on Hercules labels Aug 7, 2024
@emcbot emcbot added the CI-Hercules-Failed **Bot use only** CI testing on Hercules for this PR has failed label Aug 8, 2024
@emcbot
Copy link

emcbot commented Aug 8, 2024

Experiment C48_ATM FAILED on Hercules in Build# 29 in
/work2/noaa/stmp/CI/HERCULES/2815/RUNTESTS/EXPDIR/C48_ATM_b4d5ed66

@emcbot
Copy link

emcbot commented Aug 8, 2024

Experiment C48_S2SW FAILED on Hercules in Build# 29 in
/work2/noaa/stmp/CI/HERCULES/2815/RUNTESTS/EXPDIR/C48_S2SW_b4d5ed66

@emcbot
Copy link

emcbot commented Aug 8, 2024

Experiment C96C48_hybatmDA FAILED on Hercules in Build# 29 in
/work2/noaa/stmp/CI/HERCULES/2815/RUNTESTS/EXPDIR/C96C48_hybatmDA_b4d5ed66

@emcbot
Copy link

emcbot commented Aug 8, 2024

Experiment C96_atm3DVar FAILED on Hercules in Build# 29 in
/work2/noaa/stmp/CI/HERCULES/2815/RUNTESTS/EXPDIR/C96_atm3DVar_b4d5ed66

@emcbot emcbot added CI-Hercules-Failed **Bot use only** CI testing on Hercules for this PR has failed and removed CI-Hercules-Failed **Bot use only** CI testing on Hercules for this PR has failed labels Aug 9, 2024
@emcbot
Copy link

emcbot commented Aug 9, 2024

CI Failed on Hercules in Build# 29
Built and ran in directory /work2/noaa/stmp/CI/HERCULES/2815


Experiment C48_ATM_b4d5ed66 Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Thu Aug  8 16:57:17 CDT 2024
Experiment C48_ATM_b4d5ed66 Terminated: *FAIL*
Error logs:
/work2/noaa/stmp/CI/HERCULES/2815/RUNTESTS/COMROOT/C48_ATM_b4d5ed66/logs/2021032312/gfsfcst.log
Experiment C48_S2SW_b4d5ed66 Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Thu Aug  8 17:09:21 CDT 2024
Experiment C48_S2SW_b4d5ed66 Terminated: *FAIL*
Error logs:
/work2/noaa/stmp/CI/HERCULES/2815/RUNTESTS/COMROOT/C48_S2SW_b4d5ed66/logs/2021032312/gfsfcst.log
Experiment C96C48_hybatmDA_b4d5ed66 Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Thu Aug  8 17:39:39 CDT 2024
Experiment C96C48_hybatmDA_b4d5ed66 Terminated: *FAIL*
Error logs:
/work2/noaa/stmp/CI/HERCULES/2815/RUNTESTS/COMROOT/C96C48_hybatmDA_b4d5ed66/logs/2021122100/gfsfcst.log
Experiment C96_atm3DVar_b4d5ed66 Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Thu Aug  8 17:45:38 CDT 2024
Experiment C96_atm3DVar_b4d5ed66 Terminated: *FAIL*
Error logs:
/work2/noaa/stmp/CI/HERCULES/2815/RUNTESTS/COMROOT/C96_atm3DVar_b4d5ed66/logs/2021122100/gfsfcst.log
Experiment C48_S2SWA_gefs_b4d5ed66 Completed 1 Cycles: *SUCCESS* at Thu Aug  8 20:30:16 CDT 2024

@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA added CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules and removed CI-Hercules-Failed **Bot use only** CI testing on Hercules for this PR has failed labels Aug 9, 2024
@emcbot emcbot added CI-Hercules-Building **Bot use only** CI testing is cloning/building on Hercules CI-Hercules-Running **Bot use only** CI testing on Hercules for this PR is in-progress and removed CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules CI-Hercules-Building **Bot use only** CI testing is cloning/building on Hercules labels Aug 9, 2024
@emcbot emcbot added CI-Hercules-Passed **Bot use only** CI testing on Hercules for this PR has completed successfully and removed CI-Hercules-Running **Bot use only** CI testing on Hercules for this PR is in-progress labels Aug 9, 2024
@emcbot
Copy link

emcbot commented Aug 9, 2024

CI Passed on Hercules in Build# 30
Built and ran in directory /work2/noaa/stmp/CI/HERCULES/2815


Experiment C48_ATM_a670daf2 Completed 1 Cycles: *SUCCESS* at Fri Aug  9 13:19:10 CDT 2024
Experiment C96_atm3DVar_a670daf2 Completed 3 Cycles: *SUCCESS* at Fri Aug  9 14:50:04 CDT 2024
Experiment C48_S2SW_a670daf2 Completed 1 Cycles: *SUCCESS* at Fri Aug  9 15:32:21 CDT 2024
Experiment C96C48_hybatmDA_a670daf2 Completed 3 Cycles: *SUCCESS* at Fri Aug  9 15:32:28 CDT 2024
Experiment C48_S2SWA_gefs_a670daf2 Completed 1 Cycles: *SUCCESS* at Fri Aug  9 15:45:13 CDT 2024

@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA changed the title Jenkins shell env update for GitHub CLI executable Jenkins Pipeline Updates Aug 12, 2024
@WalterKolczynski-NOAA WalterKolczynski-NOAA merged commit d994642 into NOAA-EMC:develop Aug 13, 2024
5 checks passed
DavidHuber-NOAA added a commit to DavidHuber-NOAA/global-workflow that referenced this pull request Aug 13, 2024
…e_rocoto

* origin/develop:
  Jenkins Pipeline Updates (NOAA-EMC#2815)
  Add Gaea C5 to CI (NOAA-EMC#2814)
  Add support for forecast-only runs on AWS (NOAA-EMC#2711)
  Add fixes to products for when REPLAY IC's are used  (NOAA-EMC#2755)
  Add capability to run forecast in segments (NOAA-EMC#2795)
CoryMartin-NOAA pushed a commit to CoryMartin-NOAA/global-workflow that referenced this pull request Aug 13, 2024
Pipeline updates:
- get `gh` location on remote machine and define global bash env
variable `$GH` for GitHub CLI
- Failed cases are now displayed accordingly in the Jenkins dashboard
(see NOTE below)
-  Added the Build # in messaging for clarity when running from re-runs.
- Replaced Matrix construct for concurrency with the parallel method
that can use dynamic case lists
- With removing of the hard coded list of cases we now get list of cases
dynamically from the PR case directory
- See new look of dashboard below (has more annotations and displays
only used cases)

NOTE: **failFast** (quitting all cases on failing of one) still does not
work because it isn't quitting the running remote shells. We can make
this a configurable capability in feature request with some custom code.
The current behavior has the remaining cases continuing to run after a
FAIL label has been issued and it is incumbent of the code manager to
kill the CI job in the controller before resetting another Ready label.

---------

Co-authored-by: David Huber <[email protected]>
@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA deleted the jenkins_shell_env_update branch August 21, 2024 16:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CI/CD Issue related to CI/CD CI-Hercules-Passed **Bot use only** CI testing on Hercules for this PR has completed successfully
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants