Skip to content

Commit

Permalink
Merge branch 'main' of https://github.com/NCAR/DART into quantile_met…
Browse files Browse the repository at this point in the history
…hods
Marlee Smith committed Sep 20, 2023

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
2 parents 78b99d0 + 6eeb453 commit fc48d9d
Showing 13 changed files with 1,598 additions and 20 deletions.
26 changes: 26 additions & 0 deletions CHANGELOG.rst
Original file line number Diff line number Diff line change
@@ -22,6 +22,32 @@ individual files.

The changes are now listed with the most recent at the top.

**September 18 2023 :: Fluxnet observation converter and obs_def_rttov13_mod.f90 bug-fixes. Tag v10.8.4**

Fluxnet obs converter:

- Generates a new observation converter (Fluxnetfull_to_obs) for eddy
covariance flux tower data (carbon, water energy fluxes)
- Documentation changes made to the older, deprecated ameriflux
converter (level4_to_obs) and the broken links have been fixed
- New flux tower observation types added to accomodate the forward
operator approach for time aggregated fluxes (daily through monthly)

obs_def_rttov13_mod.f90 bug-fixes:

- Added public get_channel to obs_def_rttov13_mod.f90 to compile WRF
successfully with rttov13.
- Removed cloud_overlap (integer) from the function: get_rttov_option_logical

**August 21 2023 :: CAM-FV shell scripts. Tag v10.8.3**

Performance improvements for CAM-FV shell scripts:

- Avoid listing files if the CAM_PHIS file already exists.
- Avoid using /var/tmp
- RUNDIR defined after CIME_OUTPUT_ROOT change.
- Warn if no inflation files found.

**August 8 2023 :: MPAS-ATM constants and readthedocs fix. Tag v10.8.2**

- MPAS-ATM constants updated to MPAS v5+
2 changes: 1 addition & 1 deletion conf.py
Original file line number Diff line number Diff line change
@@ -21,7 +21,7 @@
author = 'Data Assimilation Research Section'

# The full version, including alpha/beta/rc tags
release = '10.8.2'
release = '10.8.4'
root_doc = 'index'

# -- General configuration ---------------------------------------------------
6 changes: 5 additions & 1 deletion guide/important-capabilities-dart.rst
Original file line number Diff line number Diff line change
@@ -76,7 +76,9 @@ is as follows:
+------------------------------------------------------------------------------------------------------+-------------------+-----------------------------------+
| `Aviso <https://www.aviso.altimetry.fr/en/home.html>`__: satellite derived sea surface height | Aviso | netCDF |
+------------------------------------------------------------------------------------------------------+-------------------+-----------------------------------+
| Level 4 Flux Tower data from `AmeriFlux <http://ameriflux.lbl.gov/>`__ | Ameriflux | Comma-separated text |
| Level 4 Flux Tower data | Ameriflux | Comma-separated text |
+------------------------------------------------------------------------------------------------------+-------------------+-----------------------------------+
| Ameriflux Fullset Flux Tower data from `AmeriFlux <https://ameriflux.lbl.gov/data/download-data>`__ | Ameriflux | Comma-separated text |
+------------------------------------------------------------------------------------------------------+-------------------+-----------------------------------+
| Level 2 soil moisture from `COSMOS <http://cosmos.hwr.arizona.edu/>`__ | COSMOS | Fixed-width text |
+------------------------------------------------------------------------------------------------------+-------------------+-----------------------------------+
@@ -104,6 +106,8 @@ is as follows:
+------------------------------------------------------------------------------------------------------+-------------------+-----------------------------------+
| Sea surface temperature | SST | netCDF |
+------------------------------------------------------------------------------------------------------+-------------------+-----------------------------------+
| Solar-Induced Fluorescence | SIF | netCDF |
+------------------------------------------------------------------------------------------------------+-------------------+-----------------------------------+
| Special Sensor Ultraviolet Spectrographic Imager `(SSUSI) <https://ssusi.jhuapl.edu/>`__ retrievals | SSUSI | netCDF |
+------------------------------------------------------------------------------------------------------+-------------------+-----------------------------------+
| World Ocean Database `(WOD) <http://www.nodc.noaa.gov/OC5/WOD09/pr_wod09.html>`__ | WOD | World Ocean Database packed ASCII |
9 changes: 5 additions & 4 deletions models/cam-fv/shell_scripts/cesm2_1/assimilate.csh.template
Original file line number Diff line number Diff line change
@@ -530,13 +530,14 @@ ${REMOVE} ${CAMINPUT}
${LINK} ${CASE}.cam_0001.i.${ATM_DATE_EXT}.nc ${CAMINPUT} || exit 90

# All of the .h0. files contain the same PHIS field, so we can link to any of them.

set hists = `${LIST} ${CASE}.cam_0001.h0.*.nc`
set MYSTRING = `grep cam_phis_filename input.nml`
set MYSTRING = `echo $MYSTRING | sed -e "s#[=,']# #g"`
set CAM_PHIS = $MYSTRING[2]
${REMOVE} ${CAM_PHIS}
${LINK} $hists[1] ${CAM_PHIS} || exit 100
# Listings are slow in productions runs; only do it if necessary
if (! -f ${CAM_PHIS}) then
set hists = `${LIST} ${CASE}.cam_0001.h0.*.nc`
${COPY} $hists[1] ${CAM_PHIS} || exit 100
endif

# Now, actually check the inflation settings

22 changes: 20 additions & 2 deletions models/cam-fv/shell_scripts/cesm2_1/setup_advanced
Original file line number Diff line number Diff line change
@@ -42,6 +42,13 @@
#PBS -o Test_0.bld1
#PBS -j oe
#
if ($?PBS_O_WORKDIR) then
# This avoids using /var/tmp, which can be purged by other jobs while this
# share queue job is running, which kills it.
setenv TMPDIR /glade/scratch/$USER/temp
mkdir -p $TMPDIR
endif

# ---------------------
# Purpose
# ---------------------
@@ -516,7 +523,6 @@ setenv COMP_GLC `./xmlquery COMP_GLC --value`
setenv COMP_ROF `./xmlquery COMP_ROF --value`
setenv CIMEROOT `./xmlquery CIMEROOT --value`
setenv EXEROOT `./xmlquery EXEROOT --value`
setenv RUNDIR `./xmlquery RUNDIR --value`
setenv CAM_CONFIG_OPTS `./xmlquery CAM_CONFIG_OPTS --value`

set max_tasks_per_node = `./xmlquery MAX_TASKS_PER_NODE --value`
@@ -672,6 +678,8 @@ echo "After xmlchanges for ROOTPEs ..."
./xmlchange GET_REFCASE=FALSE

./xmlchange CIME_OUTPUT_ROOT=${cime_output}
# Changing CIME_OUTPUT_ROOT changes RUNDIR in CESM, so change it here too.
setenv RUNDIR `./xmlquery RUNDIR --value`

./xmlchange SSTICE_DATA_FILENAME=$sst_dataset
./xmlchange SSTICE_GRID_FILENAME=$sst_grid
@@ -895,7 +903,7 @@ while ($inst <= $num_instances)
# echo " mfilt = 1 " >> ${fname}

echo $CAM_CONFIG_OPTS | grep 'cam4'
if ($status == 0) echo " fv_div24del2_flag = 4 " >> ${fname}
if ($status == 0) echo " fv_div24del2flag = 4 " >> ${fname}

# Settings that differ between regular CAM and the WACCM version:

@@ -1236,6 +1244,16 @@ else # CONTINUE_RUN == FALSE
echo "All files set to run the FIRST experiment step using (ref)time" $init_time

endif

ls *inf*\${restart_time}* >& /dev/null
if (\$status != 0) then
echo ""
echo '==========================================================='
echo "WARNING: no inflation files were found for \$restart_time"
echo '==========================================================='
echo ""
endif

exit 0

EndOfText
22 changes: 20 additions & 2 deletions models/cam-fv/shell_scripts/cesm2_1/setup_hybrid
Original file line number Diff line number Diff line change
@@ -42,6 +42,13 @@
#PBS -o Test_0.bld1
#PBS -j oe
#
if ($?PBS_O_WORKDIR) then
# This avoids using /var/tmp, which can be purged by other jobs while this
# share queue job is running, which kills it.
setenv TMPDIR /glade/scratch/$USER/temp
mkdir -p $TMPDIR
endif

# ---------------------
# Purpose
# ---------------------
@@ -484,7 +491,6 @@ setenv COMP_GLC `./xmlquery COMP_GLC --value`
setenv COMP_ROF `./xmlquery COMP_ROF --value`
setenv CIMEROOT `./xmlquery CIMEROOT --value`
setenv EXEROOT `./xmlquery EXEROOT --value`
setenv RUNDIR `./xmlquery RUNDIR --value`
setenv CAM_CONFIG_OPTS `./xmlquery CAM_CONFIG_OPTS --value`

set max_tasks_per_node = `./xmlquery MAX_TASKS_PER_NODE --value`
@@ -640,6 +646,8 @@ echo "After xmlchanges for ROOTPEs ..."
./xmlchange GET_REFCASE=FALSE

./xmlchange CIME_OUTPUT_ROOT=${cime_output}
# Changing CIME_OUTPUT_ROOT changes RUNDIR in CESM, so change it here too.
setenv RUNDIR `./xmlquery RUNDIR --value`

./xmlchange SSTICE_DATA_FILENAME=$sst_dataset
./xmlchange SSTICE_GRID_FILENAME=$sst_grid
@@ -860,7 +868,7 @@ while ($inst <= $num_instances)
# echo " mfilt = 1 " >> ${fname}

echo $CAM_CONFIG_OPTS | grep 'cam4'
if ($status == 0) echo " fv_div24del2_flag = 4 " >> ${fname}
if ($status == 0) echo " fv_div24del2flag = 4 " >> ${fname}

# Settings that differ between regular CAM and the WACCM version:

@@ -1128,6 +1136,16 @@ else # CONTINUE_RUN == FALSE
echo "All files set to run the FIRST experiment step using (ref)time" $init_time
endif
ls *inf*\${restart_time}* >& /dev/null
if (\$status != 0) then
echo ""
echo '==========================================================='
echo "WARNING: no inflation files were found for \$restart_time"
echo '==========================================================='
echo ""
endif
exit 0
EndOfText
5 changes: 2 additions & 3 deletions observations/forward_operators/obs_def_rttov13_mod.f90
Original file line number Diff line number Diff line change
@@ -397,7 +397,8 @@ module obs_def_rttov_mod
write_rttov_metadata, &
interactive_rttov_metadata, &
get_expected_radiance, &
get_rttov_option_logical
get_rttov_option_logical, &
get_channel

! The rttov_test.f90 program uses these, but no one else should.

@@ -4238,8 +4239,6 @@ function get_rttov_option_logical(field_name) result(p)
p = USER_CLD_OPT_PARAM
case('GRID_BOX_AVG_CLOUD')
p = GRID_BOX_AVG_CLOUD
case('CLOUD_OVERLAP')
p = cloud_overlap
case('ADDPC')
p = ADDPC
case('ADDRADREC')
1,078 changes: 1,078 additions & 0 deletions observations/obs_converters/Ameriflux/fluxnetfull_to_obs.f90

Large diffs are not rendered by default.

15 changes: 15 additions & 0 deletions observations/obs_converters/Ameriflux/fluxnetfull_to_obs.nml
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
&fluxnetfull_to_obs_nml
text_input_file = 'textdata.input',
obs_out_file = 'obs_seq.out',
timezoneoffset = -1,
latitude = -1.0,
longitude = -1.0,
elevation = -1.0,
flux_height = -1.0,
maxgoodqc = 3,
gap_filled = .true.
energy_balance = .false.
time_resolution = 'HR'
verbose = .false.
/

399 changes: 399 additions & 0 deletions observations/obs_converters/Ameriflux/fluxnetfull_to_obs.rst

Large diffs are not rendered by default.

14 changes: 8 additions & 6 deletions observations/obs_converters/Ameriflux/level4_to_obs.rst
Original file line number Diff line number Diff line change
@@ -7,12 +7,14 @@ Overview
AmeriFlux level 4 data to DART observation sequence converter
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

| This routine is designed to convert the flux tower Level 4 data from the `AmeriFlux <http://ameriflux.lbl.gov>`__
network of observations from micrometeorological tower sites. AmeriFlux is part of
`FLUXNET <http://fluxnet.ornl.gov>`__ and the converter is hoped to be a suitable starting point for the conversion of
observations from FLUXNET. As of May 2012, I have not yet tried to work with any other observations from FLUXNET.
| The AmeriFlux Level 4 products are recorded using the local time. DART observation sequence files use GMT. For more
information about AmeriFlux data products, go to http://ameriflux.lbl.gov.
| This routine was designed to convert the flux tower Level 4 data from the `AmeriFlux <http://ameriflux.lbl.gov>`__
network of observations from micrometeorological tower sites. The download link and flux data format for this code
*has been discontinued* (e.g. ``USBar2004_L4_h.txt``). Thus if you are using flux obs converters for the first time
*PLEASE USE* the updated ``Fluxnetfull_to_obs.f90`` converter and follow the documentation there :doc:`./Fluxnetfull_to_obs`
We have kept this code available if you still use the older Ameriflux data format. Also this code uses a general approach
to calculating sensible, latent and net ecosystem exchange uncertainty, that may be helpful.
| The AmeriFlux Level 4 data products are provided in the local time of the flux tower location. DART observation sequence
files are provided in UTC, thus this routine includes a time conversion.
.. warning::

19 changes: 18 additions & 1 deletion observations/obs_converters/Ameriflux/work/input.nml
Original file line number Diff line number Diff line change
@@ -27,7 +27,7 @@
text_input_file = '../data/USHa12003_L4_h.txt'
obs_out_file = 'obs_seq.out'
year = 2003
timezoneoffset = -6
timezoneoffset = -5
latitude = 42.5378
longitude = -72.1715
elevation = 353
@@ -36,6 +36,23 @@
verbose = .TRUE.
/

&fluxnetfull_to_obs_nml
text_input_file = '../data/AMF_US-Ha1_FLUXNET_FULLSET_MM_1991-2020_3-5.csv'
obs_out_file = 'obs_seq.out'
timezoneoffset = -5
latitude = 42.5378
longitude = -72.1715
elevation = 353
flux_height = 29
maxgoodqc = 3
gap_filled = .true.
energy_balance = .false.
time_resolution = 'MM'
verbose = .true.
/



# This is appropriate for a days worth of flux tower observations
# the obs in the file end 1 second before the time in the name.
# If you are using these obs with CLM, ending 1 second before is appropriate.
1 change: 1 addition & 0 deletions observations/obs_converters/Ameriflux/work/quickbuild.sh
Original file line number Diff line number Diff line change
@@ -14,6 +14,7 @@ LOCATION=threed_sphere


programs=(
fluxnetfull_to_obs
level4_to_obs
obs_sequence_tool
advance_time

0 comments on commit fc48d9d

Please sign in to comment.