DO NOT MERGE: staging of integration changes for public review

examples/run_calibration.sh

@@ -0,0 +1,45 @@
+#!/bin/bash
+# ------------------------------------------------
+# This script automates the entire calibration process
+# from generating input to executing calibration and validation runs.
+# User needs to specify basin, workdir and usrdir.
+#
+# Author: Xia Feng
+# ------------------------------------------------
+
+# Activate virtual environment
+source venv/bin/activate 
+python=`which python3`
+
+# Working Directory 
+basin='01010101'
+workdir='usrdir/working_dir/'$basin 
+
+# Create input data and files 
+pyscript1='usrdir/NextGen_Model_Calibration/python/createInput/examples/create_input.py'
+config_file_input='usrdir/NextGen_Model_Calibration/python/createInput/examples/config/input.config'
+echo 'Creating Input Data and Files'
+$python $pyscript1 $config_file_input 
+echo 'Input Data and Files are Generated'
+
+# Execute calibration run
+pyscript2='usrdir/NextGen_Model_Calibration/python/runCalibValid/calibration.py'
+config_file_calib=$workdir$'/Input/'$basin'_config_calib.yaml'
+echo 'Running Calibration Simulation'
+$python $pyscript2 $config_file_calib >>'calib_run_out'
+echo 'Calibration Run Is Completed'
+
+# Execute validation run
+pyscript3='usrdir/NextGen_Model_Calibration/python/runCalibValid/validation.py'
+config_file_control=$workdir'/Output/Validation_Run/'$basin'_config_valid_control.yaml'
+config_file_best=$workdir'/Output/Validation_Run/'$basin'_config_valid_best.yaml'
+config_file_valid_run=( $config_file_control $config_file_best )
+run_type=( 'control' 'best' )
+i=0
+while [ $i -le 1 ]
+do
+    echo 'Running Validation' ${run_type[$i]^} 'Simulation'
+    $python $pyscript3 ${config_file_valid_run[$i]} >>'valid_run_out'
+    i=$(($i+1))
+done
+echo 'Validation Run Is Completed'

python/createInput/CHANGELOG.md

@@ -0,0 +1,21 @@
+All notable changes to this project will be documented in this file.
+We follow the [Semantic Versioning 2.0.0](http://semver.org/) format.
+
+
+## x.y.z - YYYY-MM-DD
+
+### Added
+
+- Lorem ipsum dolor sit amet
+
+### Deprecated
+
+- Nothing.
+
+### Removed
+
+- Nothing.
+
+### Fixed
+
+- Nothing.

python/createInput/CONTRIBUTING.md

@@ -0,0 +1,32 @@
+# Guidance on how to contribute
+
+> All contributions to this project will be released to the public domain.
+> By submitting a pull request or filing a bug, issue, or
+> feature request, you are agreeing to comply with this waiver of copyright interest.
+> Details can be found in our [TERMS](TERMS.md) and [LICENSE](LICENSE).
+
+
+There are two primary ways to help:
+ - Using the issue tracker, and
+ - Changing the code-base.
+
+
+## Using the issue tracker
+
+Use the issue tracker to suggest feature requests, report bugs, and ask questions.
+This is also a great way to connect with the developers of the project as well
+as others who are interested in this solution.
+
+Use the issue tracker to find ways to contribute. Find a bug or a feature, mention in
+the issue that you will take on that effort, then follow the _Changing the code-base_
+guidance below.
+
+
+## Changing the code-base
+
+Generally speaking, you should fork this repository, make changes in your
+own fork, and then submit a pull request. All new code should have associated
+unit tests that validate implemented features and the presence or lack of defects.
+Additionally, the code should follow any stylistic and architectural guidelines
+prescribed by the project. In the absence of such guidelines, mimic the styles
+and patterns in the existing code-base.

python/createInput/INSTALL.md

@@ -0,0 +1,3 @@
+# Installation instructions
+
+Detailed instructions on how to install, configure, and get the project running.

python/createInput/LICENSE

@@ -0,0 +1,7 @@
+“Software code created by U.S. Government employees is not subject to copyright
+in the United States (17 U.S.C. §105). The United States/Department of Commerce
+reserve all rights to seek and obtain copyright protection in countries other
+than the United States for Software authored in its entirety by the Department
+of Commerce. To this end, the Department of Commerce hereby grants to Recipient
+a royalty-free, nonexclusive license to use, copy, and create derivative works
+of the Software outside of the United States.”

python/createInput/README.md

@@ -0,0 +1,11 @@
+## CreateInput
+
+<font size="4"> CreateInput is a Python package to generate all the necessary data and files needed for performing the model parameter calibration and validation in the NextGen modeling framework. </font>
+
+## Files in this Subpackage 
+- <font size="4">___Examples_ Directory__
+  - <font size="4">_create_input.py_: Main script to read master configuration file _input.config_ and call functions in _ginputfunc.py_ to generate the required input data and files. </font>
+  - <font size="4">_config/input.config_: This input configuration file contains all kinds of options related to the basin, calibration settings, hydrofabric file, forcing data, and formulation libraries, etc. </font>
+- <font size="4">___src_ Directory__
+  - <font size="4">_ginputfunc.py_: Contians various funcitons to create crosswalk file, generate or process BMI initial configuration files, as well as create t-route configuration file, main realization file, and calibration configuration file. </font>
+  - <font size="4">_noaa_owp.py_: Contains function to generate BMI initial configuraiton for Noah-OWP-Modular model. </font>

python/createInput/SECURITY.md

@@ -0,0 +1,27 @@
+# Security Policy
+
+There MUST be no unpatched vulnerabilities of medium or higher severity that have been publicly known for more than 60 days. 
+
+The vulnerability must be patched and released by the project itself (patches may be developed elsewhere). A vulnerability becomes publicly known (for this purpose) once it has a CVE with publicly released non-paywalled information (reported, for example, in the <a href="https://nvd.nist.gov/">National Vulnerability Database</a>) or when the project has been informed and the information has been released to the public (possibly by the project). A vulnerability is considered medium or higher severity if its <a href="https://www.first.org/cvss/" >Common Vulnerability Scoring System (CVSS)</a> base qualitative score is medium or higher. In CVSS versions 2.0 through 3.1, this is equivalent to a CVSS score of 4.0 or higher. Projects may use the CVSS score as published in a widely-used vulnerability database (such as the <a href="https://nvd.nist.gov">National Vulnerability Database</a>) using the most-recent version of CVSS reported in that database. Projects may instead calculate the severity themselves using the latest version of <a href="https://www.first.org/cvss/">CVSS</a> at the time of the vulnerability disclosure, if the calculation inputs are publicly revealed once the vulnerability is publicly known. <strong>Note</strong>: this means that users might be left vulnerable to all attackers worldwide for up to 60 days. This criterion is often much easier to meet than what Google recommends in <a href="https://security.googleblog.com/2010/07/rebooting-responsible-disclosure-focus.html">Rebooting responsible disclosure</a>, because Google recommends that the 60-day period start when the project is notified _even_ if the report is not public. Also note that this badge criterion, like other criteria, applies to the individual project. Some projects are part of larger umbrella organizations or larger projects, possibly in multiple layers, and many projects feed their results to other organizations and projects as part of a potentially-complex supply chain. An individual project often cannot control the rest, but an individual project can work to release a vulnerability patch in a timely way. Therefore, we focus solely on the individual project's response time.  Once a patch is available from the individual project, others can determine how to deal with the patch (e.g., they can update to the newer version or they can apply just the patch as a cherry-picked solution).
+
+The public repositories MUST NOT leak any valid private credential (e.g., a working password or private key) that is intended to limit public access.
+
+## Supported Versions
+
+Use this section to tell people about which versions of your project are
+currently being supported with security updates.
+
+| Version | Supported          |
+| ------- | ------------------ |
+| 5.1.x   | :white_check_mark: |
+| 5.0.x   | :x:                |
+| 4.0.x   | :white_check_mark: |
+| < 4.0   | :x:                |
+
+## Reporting a Vulnerability
+
+Use this section to tell people how to report a vulnerability.
+
+Tell them where to go, how often they can expect to get an update on a
+reported vulnerability, what to expect if the vulnerability is accepted or
+declined, etc.

python/createInput/TERMS.md

@@ -0,0 +1,19 @@
+## Disclaimer
+
+This repository is a scientific product and is not official communication of the National Oceanic and Atmospheric Administration, or the United States Department of Commerce. All NOAA GitHub project code is provided on an 'as is' basis and the user assumes responsibility for its use. Any claims against the Department of Commerce or Department of Commerce bureaus stemming from the use of this GitHub project will be governed by all applicable Federal law. Any reference to specific commercial products, processes, or services by service mark, trademark, manufacturer, or otherwise, does not constitute or imply their endorsement, recommendation or favoring by the Department of Commerce. The Department of Commerce seal and logo, or the seal and logo of a DOC bureau, shall not be used in any manner to imply endorsement of any commercial product or activity by DOC or the United States Government.
+
+[NOAA GitHub Policy](https://github.com/NOAAGov/Information)
+
+### Other Information
+
+Unless expressly stated otherwise, the person who associated a work with
+this deed makes no warranties about the work, and disclaims liability for
+all uses of the work, to the fullest extent permitted by applicable law.
+When using or citing the work, you should not imply endorsement by the
+author or the affirmer.
+
+## Exceptions
+
+_Source code or other assets that are excluded from the TERMS should be listed
+here. These may include dependencies that may be licensed differently or are
+not in the public domain._

python/createInput/examples/config/input.config

@@ -0,0 +1,187 @@
+# ------------------------------------------------
+# NextGen Model Calibration Configuration File
+#
+# This file is used to create the input data and
+# files to run calibration.
+# It contains three sections:
+#    1. General
+#    2. Calibration
+#    3. DataFile
+# User needs to fill out the required entries of each
+# section for the specified basin.
+#
+# @author: Xia Feng
+# ------------------------------------------------
+
+[General]
+# Stream gage ID at basin outlet 
+basin = 01010101
+
+# Model and module for calibration
+model = cfe
+
+# Calibration or validation run
+run_type = calib
+
+# Main directory to store input, output and other files  
+main_dir = 
+
+[Calibration]
+# Optimizaition algorithm for parameter estimation
+optimization_algorithm = dds
+
+# Specify algorithm parameters for PSO and GWO only
+# Control parameters c1, c2 and w are only needed for PSO
+swarm_size = 20
+c1 = 2
+c2 = 2
+w = 0.7
+
+# Objective function   
+objective_function = kge
+
+# Starting iteration number
+start_iteration = 0
+
+# Number of iterations
+number_iteration = 10
+
+# Whether restart calibration from the stopped iteration 
+# 0: Not 
+# 1: Yes
+# It should be 0 if start_interation entry is 0. 
+restart = 0
+
+# Calibration time period
+calib_start_period = 2019-10-01 00:00:00
+calib_end_period = 2022-10-01 00:00:00
+
+# Calibration evaluation time period
+calib_eval_start_period = 2020-10-01 00:00:00
+calib_eval_end_period = 2022-10-01 00:00:00
+
+# Validation time period
+# These entries are optional.
+# If they are not specified, validation run will not be performed.
+valid_start_period = 2016-10-01 00:00:00
+valid_end_period = 2019-10-01 00:00:00
+
+# Validation evaluation time period
+# These entries are optional.
+# If validation run is not performed, there is no need to fill these two options.
+valid_eval_start_period = 2017-10-01 00:00:00
+valid_eval_end_period = 2019-10-01 00:00:00
+
+# Full evaluation time period
+# These entries are optional.
+# If validation run is not performed, there is no need to fill these two options.
+full_eval_start_period = 2016-10-01 00:00:00
+full_eval_end_period = 2022-10-01 00:00:00
+
+# Save streamflow output and plot at the specified iteration
+# These entries are optional and specified with the default values.
+# 1: Filename is distinguished by the iteration number.
+# 0: Filename is same at different iteration, i.e., overwrite file from the last iteration.
+save_plot_iter = 0
+save_output_iter = 0
+
+# Iteration interval to save plots
+# This entry is optional and specified with the default value.
+save_plot_iter_freq = 50
+
+# Streamflow threshold in cms for the calculation of categorical scores
+# This entry is optional. 
+# If it is not specified, categorical metrics will not be calculated.
+streamflow_threshold = 
+
+# Stream station name used for the title of plots 
+# This entry is optional. 
+# If it is not specified, station name will not show up in the title of plots.
+station_name =  
+
+# Email address to receive the notification of run completion 
+# This entry is optional. 
+# If it is not specified, user will not receive email notification after run is completed. 
+user_email = [email protected]
+
+[DataFile]
+# Diretory for forcing data 
+forcing_dir = 
+
+# Diretory for streamflow observation  
+# This entry is optional. 
+# If it is not specified, observation will be downloaded on the fly during the calibration.
+obs_dir = 
+
+# Diretory for hydrofabric data   
+hydrofab_dir = 
+
+# Diretory for CFE BMI files 
+# This entry is optional.
+# If CFE is not selected, there is no need to fill this entry.
+cfe_dir = 
+
+# Diretory for TOPMODEL BMI files   
+# This entry is optional.
+# If TOPMODEL is not selected, there is no need to fill this entry.
+topmd_dir = 
+
+# Diretory for Noah-OWP-Modular parameter table   
+# This entry is optional.
+# If Noah-OWP-Modular is not selected, there is no need to fill this entry.
+noah_parameter_dir = 
+
+# Path for model attributes file
+# This entry is optional.
+# If neither CFE, nor Noah-OWP-Modular, nor SFT is selected, there is no need to fill this entry.
+attributes_file = 
+
+# Path for calibration parameter file
+# It contains the minimum, maximum, and initial values of parameters for the specified model
+calib_parameter_file = 
+
+# Path for LASAM soil parameter file
+# This entry is optional.
+# If LASAM is not selected, there is no need to fill this entry.
+lasam_soil_parameter_file = 
+
+# Path for LASAM soil class file
+# This entry is optional.
+# If LASAM is not selected, there is no need to fill this entry.
+lasam_soil_class_file = 
+
+# Executable file for running ngen BMI
+ngen_exe_file = 
+
+# Library file for CFE
+# This entry is optional.
+# If CFE is not selected, there is no need to fill this entry.
+cfe_lib = 
+
+# Library file for SLoth
+sloth_lib = 
+
+# Library file for TOPMODEL
+# This entry is optional.
+# If TOPMODEL is not selected, there is no need to fill this entry.
+topmd_lib = 
+
+# Library file for Noah-OWP-Modular
+# This entry is optional.
+# If Noah-OWP-Modular is not selected, there is no need to fill this entry.
+noah_lib = 
+
+# Library file for soil freeze and thaw (SFT) model
+# This entry is optional.
+# If SFT is not selected, there is no need to fill this entry.
+sft_lib = 
+
+# Library file for soil moisture profiles (SMP)
+# This entry is optional.
+# If SMP is not selected, there is no need to fill this entry.
+smp_lib = 
+
+# Library file for Lumped Arid and Semi-arid Model (LASAM)
+# This entry is optional.
+# If LASAM is not selected, there is no need to fill this entry.
+lasam_lib =