Eliminate race conditions and remove DATAROOT last in cleanup

[DavidHuber-NOAA]

Description

This changes the order of the cleanup job so that the working directory is deleted at the end. It also adds the -ignore_readdir_race flag to find to prevent errors if a file was deleted after the list of files was collected. This can happen if two consecutive cycles run the cleanup job at the same time.

Resolves #2880

Type of change

• Bug fix (fixes something broken)

Change characteristics

• Is this a breaking change (a change in existing functionality)? NO
• Does this change require a documentation update? NO
• Does this change require an update to any of the following submodules? NO

How has this been tested?

A 5-cycle test on WCOSS2

Checklist

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• My changes generate no new warnings
• New and existing tests pass with my changes

[aerorahul]

Looks good.
I found a suggestion online to use sync that ensures all modifications are synchronized.
It should help at the least, but look it up and validate it.

[XuanliLi-NOAA]

Changes look good to me. I had removed the files in my directories due to the disk quota limit, but Sean Casey at QOSAP tested the changes (including those suggested by @aerorahul) on Hera and confirmed that they solved the problem.

[emcbot]

CI Update on Wcoss2 at 09/05/24 08:40:17 PM
=================================================
PR:2893 Reset to Wcoss2-Ready by user and is now restarting CI tests
No current experiments to cancel in PR: 2893 on Wcoss2

[emcbot]

CI Update on Wcoss2 at 09/05/24 08:40:21 PM
============================================
Cloning and Building global-workflow PR: 2893
with PID: 146574 on host: dlogin03

[emcbot]

Automated global-workflow Testing Results:

Machine: Wcoss2
Start: Thu Sep  5 20:43:11 UTC 2024 on dlogin03
---------------------------------------------------
Build: Completed at 09/05/24 09:27:58 PM
Case setup: Completed for experiment C48_ATM_b502db38
Case setup: Skipped for experiment C48mx500_3DVarAOWCDA_b502db38
Case setup: Skipped for experiment C48_S2SWA_gefs_b502db38
Case setup: Completed for experiment C48_S2SW_b502db38
Case setup: Completed for experiment C96_atm3DVar_extended_b502db38
Case setup: Skipped for experiment C96_atm3DVar_b502db38
Case setup: Completed for experiment C96C48_hybatmaerosnowDA_b502db38
Case setup: Completed for experiment C96C48_hybatmDA_b502db38
Case setup: Completed for experiment C96C48_ufs_hybatmDA_b502db38

[emcbot]

All CI Test Cases Passed on Wcoss2:

Experiment C48_ATM_b502db38 *** SUCCESS *** at 09/05/24 10:57:10 PM
Experiment C48_S2SW_b502db38 *** SUCCESS *** at 09/05/24 11:00:17 PM
Experiment C96C48_hybatmDA_b502db38 *** SUCCESS *** at 09/06/24 12:06:41 AM
Experiment C96C48_hybatmaerosnowDA_b502db38 *** SUCCESS *** at 09/06/24 01:03:33 AM
Experiment C96C48_ufs_hybatmDA_b502db38 *** SUCCESS *** at 09/06/24 01:51:25 AM
Experiment C96_atm3DVar_extended_b502db38 *** SUCCESS *** at 09/06/24 10:42:49 AM