Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix convective_transition_diag POD #695

Merged
merged 8 commits into from
Oct 2, 2024
Merged

Fix convective_transition_diag POD #695

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
e559810
fix ctd file formatting and typos
wrongkindofdoctor Oct 1, 2024
1ce5955
more formatting and typo fixes in ctd POD
wrongkindofdoctor Oct 1, 2024
ee6a410
uncomment convective transistion diag POD in 1a CI test config files
wrongkindofdoctor Oct 1, 2024
43ed478
try moving convective_transition_pod to ubuntu suite 2 tests
wrongkindofdoctor Oct 1, 2024
c96c9bb
add wkdir cleanup between each test run step and separate obs data fe…
wrongkindofdoctor Oct 2, 2024
e4524ec
move convective_transition_diag POD to set 1b tests
wrongkindofdoctor Oct 2, 2024
31dfc88
just run 1 POD in set 1a and 2 PODs in set 1b to avoid runner timeouts
wrongkindofdoctor Oct 2, 2024
3120a9a
reorganize 1b tests
wrongkindofdoctor Oct 2, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 27 additions & 14 deletions .github/workflows/mdtf_tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,7 @@ jobs:
mkdir wkdir
## make input data directories
mkdir -p inputdata/obs_data
- name: Get Observational Data for Set 1
- name: Get Observational Data for Set 1a
run: |
echo "${PWD}"
cd ../
Expand All @@ -106,18 +106,12 @@ jobs:
# attempt FTP data fetch
# allow 20 min for transfer before timeout; Github actions allows 6 hours for individual
# jobs, but we don't want to max out resources that are shared by the NOAA-GFDL repos.
curl --verbose --ipv4 --connect-timeout 8 --max-time 1200 --retry 128 --ftp-ssl --ftp-pasv -u "anonymous:anonymous" ftp://ftp.gfdl.noaa.gov/perm/oar.gfdl.mdtf/convective_transition_diag_obs_data.tar --output convective_transition_diag_obs_data.tar
curl --verbose --ipv4 --connect-timeout 8 --max-time 1200 --retry 128 --ftp-ssl --ftp-pasv -u "anonymous:anonymous" ftp://ftp.gfdl.noaa.gov/perm/oar.gfdl.mdtf/EOF_500hPa_obs_data.tar --output EOF_500hPa_obs_data.tar
# curl --verbose --ipv4 --connect-timeout 8 --max-time 1200 --retry 128 --ftp-ssl --ftp-pasv -u "anonymous:anonymous" ftp://ftp.gfdl.noaa.gov/perm/oar.gfdl.mdtf/EOF_500hPa_obs_data.tar --output EOF_500hPa_obs_data.tar
curl --verbose --ipv4 --connect-timeout 8 --max-time 1200 --retry 128 --ftp-ssl --ftp-pasv -u "anonymous:anonymous" ftp://ftp.gfdl.noaa.gov/perm/oar.gfdl.mdtf/Wheeler_Kiladis_obs_data.tar --output Wheeler_Kiladis_obs_data.tar
curl --verbose --ipv4 --connect-timeout 8 --max-time 1200 --retry 128 --ftp-ssl --ftp-pasv -u "anonymous:anonymous" ftp://ftp.gfdl.noaa.gov/perm/oar.gfdl.mdtf/MJO_teleconnection_obs_data.tar --output MJO_teleconnection_obs_data.tar
curl --verbose --ipv4 --connect-timeout 8 --max-time 1200 --retry 128 --ftp-ssl --ftp-pasv -u "anonymous:anonymous" ftp://ftp.gfdl.noaa.gov/perm/oar.gfdl.mdtf/MJO_suite_obs_data.tar --output MJO_suite_obs_data.tar
curl --verbose --ipv4 --connect-timeout 8 --max-time 1200 --retry 128 --ftp-ssl --ftp-pasv -u "anonymous:anonymous" ftp://ftp.gfdl.noaa.gov/perm/oar.gfdl.mdtf/precip_diurnal_cycle_obs_data.tar --output precip_diurnal_cycle_obs_data.tar
echo "Untarring set 1 NCAR/CESM standard test files"
tar -xvf convective_transition_diag_obs_data.tar
tar -xvf EOF_500hPa_obs_data.tar
echo "Untarring set 1a NCAR/CESM standard test files"
# tar -xvf EOF_500hPa_obs_data.tar
tar -xvf precip_diurnal_cycle_obs_data.tar
tar -xvf MJO_teleconnection_obs_data.tar
tar -xvf MJO_suite_obs_data.tar
tar -xvf Wheeler_Kiladis_obs_data.tar
# clean up tarballs
rm -f *.tar
Expand All @@ -133,6 +127,23 @@ jobs:
./mdtf -f ${{matrix.json-file-1a}}
# Debug POD log(s)
# cat ${POD_OUTPUT}/MDTF_NCAR.Synthetic_1975_1981/Wheeler_Kiladis/Wheeler_Kiladis.log
- name: Get observational data for set 1b
run: |
# clean up data from previous runs
echo "deleting data from set 1a"
cd ../wkdir
rm -rf *
cd ../inputdata/obs_data
rm -rf *
cd ../../
curl --verbose --ipv4 --connect-timeout 8 --max-time 1200 --retry 128 --ftp-ssl --ftp-pasv -u "anonymous:anonymous" ftp://ftp.gfdl.noaa.gov/perm/oar.gfdl.mdtf/convective_transition_diag_obs_data.tar --output convective_transition_diag_obs_data.tar
curl --verbose --ipv4 --connect-timeout 8 --max-time 1200 --retry 128 --ftp-ssl --ftp-pasv -u "anonymous:anonymous" ftp://ftp.gfdl.noaa.gov/perm/oar.gfdl.mdtf/MJO_teleconnection_obs_data.tar --output MJO_teleconnection_obs_data.tar
curl --verbose --ipv4 --connect-timeout 8 --max-time 1200 --retry 128 --ftp-ssl --ftp-pasv -u "anonymous:anonymous" ftp://ftp.gfdl.noaa.gov/perm/oar.gfdl.mdtf/MJO_suite_obs_data.tar --output MJO_suite_obs_data.tar
tar -xvf MJO_teleconnection_obs_data.tar
tar -xvf MJO_suite_obs_data.tar
tar -xvf convective_transition_diag_obs_data.tar
# clean up tarballs
rm -f *.tar
- name: Run diagnostic tests set 1b
run: |
./mdtf -f ${{matrix.json-file-1b}}
Expand All @@ -141,8 +152,10 @@ jobs:
echo "${PWD}"
# remove data from previous run
# Actions moves you to the root repo directory in every step, so need to cd again
echo "deleting data from set 1b"
cd ../wkdir
rm -rf *
cd ../inputdata/obs_data
echo "deleting obs data from set 1"
rm -rf *
cd ../../
echo "Available Space"
Expand All @@ -157,8 +170,6 @@ jobs:
rm -f *.tar
- name: Run diagnostic tests set 2
run: |
micromamba activate _MDTF_base
# run the test PODs
./mdtf -f ${{matrix.json-file-2}}
# Uncomment the following line for debugging
#cat ../wkdir/MDTF_GFDL.Synthetic_1_10/MJO_prop_amp/MJO_prop_amp.log
Expand All @@ -167,8 +178,10 @@ jobs:
echo "${PWD}"
# remove data from previous run
# Actions moves you to the root repo directory in every step, so need to cd again
echo "deleting data from set 2"
cd ../wkdir
rm -rf *
cd ../inputdata/obs_data
echo "deleting obs data from set 2"
rm -rf *
cd ../../
echo "Available Space"
Expand Down
13 changes: 7 additions & 6 deletions diagnostics/convective_transition_diag/convecTransBasic.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,7 @@
from convecTransBasic_util import convecTransBasic_calc_model
from convecTransBasic_util import convecTransBasic_loadAnalyzedData
from convecTransBasic_util import convecTransBasic_plot

print("**************************************************")
print("Excuting Convective Transition Basic Statistics (convecTransBasic.py)......")
print("**************************************************")
Expand All @@ -77,8 +78,8 @@
print("Load user-specified binning parameters..."),

# Create and read user-specified parameters
os.system("python "+ os.environ["POD_HOME"]+ "/" + "convecTransBasic_usp_calc.py")
with open(os.environ["WORK_DIR"]+"/" + "convecTransBasic_calc_parameters.json") as outfile:
os.system("python " + os.environ["POD_HOME"] + "/" + "convecTransBasic_usp_calc.py")
with open(os.environ["WORK_DIR"] + "/" + "convecTransBasic_calc_parameters.json") as outfile:
bin_data = json.load(outfile)
print("...Loaded!")

Expand Down Expand Up @@ -108,15 +109,15 @@
+ ") will be saved to " + bin_data["PREPROCESSING_OUTPUT_DIR"] + "/")

# Load & pre-process region mask
REGION=generate_region_mask(bin_data["REGION_MASK_DIR"] + "/" + bin_data["REGION_MASK_FILENAME"],
bin_data["pr_list"][0], bin_data["LAT_VAR"], bin_data["LON_VAR"])
REGION = generate_region_mask(bin_data["REGION_MASK_DIR"] + "/" + bin_data["REGION_MASK_FILENAME"],
bin_data["pr_list"][0], bin_data["LAT_VAR"], bin_data["LON_VAR"])

# Pre-process temperature (if necessary) & bin & save binned results
binned_output=convecTransBasic_calc_model(REGION, bin_data["args1"])
binned_output = convecTransBasic_calc_model(REGION, bin_data["args1"])

else: # Binned data file exists & BIN_ANYWAY=False
print("Binned output detected..."),
binned_output=convecTransBasic_loadAnalyzedData(bin_data["args2"])
binned_output = convecTransBasic_loadAnalyzedData(bin_data["args2"])
print("...Loaded!")

# ======================================================================
Expand Down
4 changes: 2 additions & 2 deletions diagnostics/convective_transition_diag/convecTransBasic_usp_plot.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
import glob

with open(os.environ["WORK_DIR"] + "/" + "convecTransBasic_calc_parameters.json") as outfile:
bin_data=json.load(outfile)
bin_data = json.load(outfile)

# ======================================================================
# START USER SPECIFIED SECTION
Expand Down Expand Up @@ -174,7 +174,7 @@
bin_data["BULK_TROPOSPHERIC_TEMPERATURE_MEASURE"]
]

data["args4"] = [ bin_data["CWV_BIN_WIDTH"], PDF_THRESHOLD, CWV_RANGE_THRESHOLD,
data["args4"] = [bin_data["CWV_BIN_WIDTH"], PDF_THRESHOLD, CWV_RANGE_THRESHOLD,
CP_THRESHOLD, bin_data["MODEL"], bin_data["REGION_STR"], bin_data["NUMBER_OF_REGIONS"],
bin_data["BULK_TROPOSPHERIC_TEMPERATURE_MEASURE"], bin_data["PRECIP_THRESHOLD"],
FIG_OUTPUT_DIR, FIG_OUTPUT_FILENAME,
Expand Down
Loading
Loading