From 4c57f59a4b6c8e58c96d392536b17c63a552eba6 Mon Sep 17 00:00:00 2001 From: bikegeek <3753118+bikegeek@users.noreply.github.com> Date: Mon, 21 Oct 2024 12:08:39 -0600 Subject: [PATCH 01/57] Delete METreformat/test/eclv_mpr_pct_vl1l2_vcnt.xml no longer used --- METreformat/test/eclv_mpr_pct_vl1l2_vcnt.xml | 14 -------------- 1 file changed, 14 deletions(-) delete mode 100644 METreformat/test/eclv_mpr_pct_vl1l2_vcnt.xml diff --git a/METreformat/test/eclv_mpr_pct_vl1l2_vcnt.xml b/METreformat/test/eclv_mpr_pct_vl1l2_vcnt.xml deleted file mode 100644 index 5ab0eb60..00000000 --- a/METreformat/test/eclv_mpr_pct_vl1l2_vcnt.xml +++ /dev/null @@ -1,14 +0,0 @@ - - ./data/point_stat/eclv_mpr_pct_vl1l2_vcnt - true - - - - grid_stat - point_stat - - - - MET output from grid-stat including the eclv, mpr, pct, Vl1l2, and vcnt line types - - From 17a1cf2a446a31a307a98c8786f45ebca164e85a Mon Sep 17 00:00:00 2001 From: bikegeek <3753118+bikegeek@users.noreply.github.com> Date: Mon, 21 Oct 2024 12:08:58 -0600 Subject: [PATCH 02/57] Delete METreformat/test/ens_stat_ecnt.xml no longer used --- METreformat/test/ens_stat_ecnt.xml | 14 -------------- 1 file changed, 14 deletions(-) delete mode 100644 METreformat/test/ens_stat_ecnt.xml diff --git a/METreformat/test/ens_stat_ecnt.xml b/METreformat/test/ens_stat_ecnt.xml deleted file mode 100644 index 8d8152df..00000000 --- a/METreformat/test/ens_stat_ecnt.xml +++ /dev/null @@ -1,14 +0,0 @@ - - ./data/ensemble_stat/ - true - - - - grid_stat - point_stat - - - - MET output from ensemble stat including the ecnt, rhist, phist, relp, ssvar, and orank line types - - From 8a44bafd0e72ea68e7c8c0366c1479038be6ee72 Mon Sep 17 00:00:00 2001 From: bikegeek <3753118+bikegeek@users.noreply.github.com> Date: Mon, 21 Oct 2024 12:09:15 -0600 Subject: [PATCH 03/57] Delete METreformat/test/mctc_mcts.xml no longer used --- METreformat/test/mctc_mcts.xml | 14 -------------- 1 file changed, 14 deletions(-) delete mode 100644 METreformat/test/mctc_mcts.xml diff --git a/METreformat/test/mctc_mcts.xml b/METreformat/test/mctc_mcts.xml deleted file mode 100644 index 4d60fe00..00000000 --- a/METreformat/test/mctc_mcts.xml +++ /dev/null @@ -1,14 +0,0 @@ - - ./data/grid_stat/mctc_mcts - true - - - - grid_stat - point_stat - - - - MET output from grid-stat including the mctc and mcts line types - - From c987f915093b354df225732fd5dbcccea6cb00a0 Mon Sep 17 00:00:00 2001 From: bikegeek <3753118+bikegeek@users.noreply.github.com> Date: Mon, 21 Oct 2024 12:09:32 -0600 Subject: [PATCH 04/57] Delete METreformat/test/point_stat.xml no longer used --- METreformat/test/point_stat.xml | 18 ------------------ 1 file changed, 18 deletions(-) delete mode 100644 METreformat/test/point_stat.xml diff --git a/METreformat/test/point_stat.xml b/METreformat/test/point_stat.xml deleted file mode 100644 index 21dca1c6..00000000 --- a/METreformat/test/point_stat.xml +++ /dev/null @@ -1,18 +0,0 @@ - - ./data/point_stat - true - - - - ensemble_stat - grid_stat - mode - point_stat - stat_analysis - wavelet_stat - - - - MET output - - From a52eef6446a26992b475bfe4952a41355aadb7fa Mon Sep 17 00:00:00 2001 From: bikegeek Date: Mon, 21 Oct 2024 15:01:21 -0600 Subject: [PATCH 05/57] Initial updates for detailed information on loading data. --- docs/Users_Guide/index.rst | 2 +- docs/Users_Guide/load_data.rst | 12 ++++++++---- docs/index.rst | 3 ++- 3 files changed, 11 insertions(+), 6 deletions(-) diff --git a/docs/Users_Guide/index.rst b/docs/Users_Guide/index.rst index 1a1b482f..8e86f7dc 100644 --- a/docs/Users_Guide/index.rst +++ b/docs/Users_Guide/index.rst @@ -73,10 +73,10 @@ Foundation (NSF). overview release-notes installation - common load_data read_nc reformat_stat_data + Indices and tables diff --git a/docs/Users_Guide/load_data.rst b/docs/Users_Guide/load_data.rst index 0dd2e509..572057e2 100644 --- a/docs/Users_Guide/load_data.rst +++ b/docs/Users_Guide/load_data.rst @@ -2,13 +2,17 @@ Background ********** -The METdbLoad module provides support for inserting MET output data into the database. +The METdbLoad module provides support for inserting MET output data into a relational database +(mysql, mariadb, or aurora). -Before using the METdbLoad module, the database must exist and have the proper permissions -(i.e. grant privileges to insert, delete, update, and index). A schema file, *mv_mysql.sql* is available in the -METdataio/METdbLoad/sql/ directory for importing into the database. +Before using the METdbLoad module, the database **must** exist and have the proper permissions +(i.e. grant privileges to insert, delete, update, and index). Next, use *mv_mysql.sql* and the mysql command +line client (https://dev.mysql.com/doc/refman/en/mysql.html) to create the tables corresponding to the MET line types. +The *mv_mysql.sql* file is located in the METdataio/METdbLoad/sql/ directory. The METdbLoad script *met_db_load.py* performs the loading of data based on settings in an XML specification file. +The XML specification file contains database connection information, the location of data to be loaded, and other +settings relevant to the type of data that is being loaded. In the METdataio/METdbLoad/sql/scripts directory, there are two configuration files: diff --git a/docs/index.rst b/docs/index.rst index 96d6738f..9d6e703e 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -96,7 +96,7 @@ Acronyms Authors ------- -Many authors, listed below in alphabetical order, have contributed to the documentation of METplus. +Many authors, listed below, have contributed to the documentation of METplus. To cite this documentation in publications, please refer to the METdataio User's Guide :ref:`Citation Instructions`. * Venita Hagerty [#CIRA]_ @@ -104,6 +104,7 @@ To cite this documentation in publications, please refer to the METdataio User's * Tatiana Burek [#NCAR]_ * Hank Fisher [#NCAR]_ + .. rubric:: Organization .. [#NCAR] `National Center for Atmospheric Research, Research From 33b4b59e827337750d2bebaea3823528c1f7e2b0 Mon Sep 17 00:00:00 2001 From: bikegeek Date: Mon, 21 Oct 2024 15:02:52 -0600 Subject: [PATCH 06/57] Initial version-needs cleaning up --- .../Examples/example_load_specification.xml | 67 +++++++++++++++++++ 1 file changed, 67 insertions(+) create mode 100644 METdbLoad/test/Examples/example_load_specification.xml diff --git a/METdbLoad/test/Examples/example_load_specification.xml b/METdbLoad/test/Examples/example_load_specification.xml new file mode 100644 index 00000000..d67ee74f --- /dev/null +++ b/METdbLoad/test/Examples/example_load_specification.xml @@ -0,0 +1,67 @@ + + + your-host:### + your-database-name + your-dbusername + your-dbpassword + + + + 2022050100 + 2022051200 + 86400 + yyyyMMddHH + + + 2022050100 + 2022051200 + 0600 + yyyyMMddHH + + + false + 1 + False + false + FALSE + true + True + false + false + Regional Ensemble + + /your-path-to/{config}/{fcst_init}/{mem}/metprd/{met_out}/ + + + + + + + + + + HREF_lag_offset + RTPS + + + mem01 + mem02 + mem03 + mem04 + mem05 + mem06 + mem07 + mem08 + mem09 + mem10 + + + grid_stat_cmn + point_stat_cmn + + + + true + Load HREF and RTPS data for Spring 2022. + + From 58a559aad170a4480f98acbb56f590eaaaa9f4f8 Mon Sep 17 00:00:00 2001 From: bikegeek Date: Mon, 21 Oct 2024 15:31:40 -0600 Subject: [PATCH 07/57] increas numbered from 4 to 6 --- docs/Users_Guide/index.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Users_Guide/index.rst b/docs/Users_Guide/index.rst index 8e86f7dc..40b977f7 100644 --- a/docs/Users_Guide/index.rst +++ b/docs/Users_Guide/index.rst @@ -68,7 +68,7 @@ Foundation (NSF). .. toctree:: :titlesonly: - :numbered: 4 + :numbered: 6 overview release-notes From e9244da321d857f8a7cb8cb87d2e23ccc05f470e Mon Sep 17 00:00:00 2001 From: bikegeek <3753118+bikegeek@users.noreply.github.com> Date: Mon, 21 Oct 2024 15:51:12 -0600 Subject: [PATCH 08/57] Delete docs/Users_Guide/common.rst outdated information, replacing XML specification information in the load_data.rst file --- docs/Users_Guide/common.rst | 28 ---------------------------- 1 file changed, 28 deletions(-) delete mode 100644 docs/Users_Guide/common.rst diff --git a/docs/Users_Guide/common.rst b/docs/Users_Guide/common.rst deleted file mode 100644 index b53e033e..00000000 --- a/docs/Users_Guide/common.rst +++ /dev/null @@ -1,28 +0,0 @@ -.. _common: - -********************* -Common XML Structures -********************* - -METdbload and the batch plotting module both use XML input files. Some XML -structures can be used in either input file. These common structures are -documented below. - -| **:** Information about the database connection. -| -| **:** Database hostname and port number in [host]:[port] format. -| **:** One or more existing database names that start with \'\mv_\'\ separated by commas. Only one name for loading. -| **:** Database user. -| **:** Database user's password. -| **:** Database type. Can be mysql, mariadb, or aurora. -| -| **** -| -| **:** A structure that specifies a list of date strings in a certain format. Has a name attribute. -| -| **:** The start date and time of the date list, specified in the format given by the tag **** -| **:** The end date and time, specified in the format given by the tag **** -| **:** The increment, in seconds, between successive members of the date list. -| **:** The date format, specified by the java class SimpleDateFormat, example yyyyMMddHH. -| -| **** From 1840cb83f9d18167867abdd6f1866c19e6f661d6 Mon Sep 17 00:00:00 2001 From: bikegeek Date: Mon, 21 Oct 2024 16:05:15 -0600 Subject: [PATCH 09/57] remove any links to common.rst --- docs/Users_Guide/load_data.rst | 3 --- 1 file changed, 3 deletions(-) diff --git a/docs/Users_Guide/load_data.rst b/docs/Users_Guide/load_data.rst index 572057e2..e938aaf3 100644 --- a/docs/Users_Guide/load_data.rst +++ b/docs/Users_Guide/load_data.rst @@ -143,9 +143,6 @@ into the database to METdbload. It is an XML file whose top-level tag is and it contains the following elements, divided into functional sections: - * **:** Please reference the :numref:`common` documentation. - - * **:** Please reference the :numref:`common` documentation. * **:** **TRUE** or **FALSE**, this option indicates whether or not to load STAT data. From 62d8b67336f448dd6588680f32be94455418840b Mon Sep 17 00:00:00 2001 From: bikegeek Date: Tue, 22 Oct 2024 18:56:37 -0600 Subject: [PATCH 10/57] Work in progress, need to add more content --- docs/Users_Guide/load_data.rst | 277 +++++++++++++++++++++++++++------ 1 file changed, 227 insertions(+), 50 deletions(-) diff --git a/docs/Users_Guide/load_data.rst b/docs/Users_Guide/load_data.rst index e938aaf3..89042e04 100644 --- a/docs/Users_Guide/load_data.rst +++ b/docs/Users_Guide/load_data.rst @@ -10,100 +10,277 @@ Before using the METdbLoad module, the database **must** exist and have the prop line client (https://dev.mysql.com/doc/refman/en/mysql.html) to create the tables corresponding to the MET line types. The *mv_mysql.sql* file is located in the METdataio/METdbLoad/sql/ directory. -The METdbLoad script *met_db_load.py* performs the loading of data based on settings in an XML specification file. +The METdbLoad script *met_db_load.py* performs loading of data based on settings in an XML specification file. The XML specification file contains database connection information, the location of data to be loaded, and other -settings relevant to the type of data that is being loaded. +settings relevant to the type of data that is being loaded. The XML specification is validated against a schema to check +that the file is valid. This validation is necessary to prevent extremely large payloads or recursive +payloads that can compromise the loading of data. +The elements in the XML specification file **must** adhere to the **order specified** by the +XML schema and conform to size and number of element limitations. -In the METdataio/METdbLoad/sql/scripts directory, there are two configuration files: +.. dropdown:: This is an **example XML specification file** that is **valid**: - * db_load_specification.xml + .. literalinclude:: ../../METdbLoad/test/Examples/example_load_specification.xml - * data_loading_config.yaml -The *db_load_specification.xml* is a template XML specification file, and *data_loading_config.yaml* -is a template YAML configuration file. The *data_loading_config.yaml* file contains -information about the database (username, password, database name, etc.). This information is used by the -*generate_xml_spec.py* script to generate the XML specification file which is then used to load data into the database. -Generate the XML Specification File +.. dropdown:: This is the **XML schema** that is used to **validate the XML specification file**: + + .. literalinclude:: ../../METdbLoad/ush/load_specification_schema.xsd + +Create the XML Specification File =================================== -Copy the *data_loading_config.yaml* file to a secure location in your workspace, as this file will contain the username -and password to the database. **Do not put this file where it can be read by anyone who should not have access to this -information.** +The XML specification file contains the database connection information, data file location, and instructions for +loading the data into the database. + + +Create your own XML specification file by copying the example specification file +*METdataio/METdbLoad/test/Examples/example_load_specification.xml* file to a +location in your workspace. This file will contain the username and password to the database. + +**Do not put this XML specification file where it can be read by anyone who should not have access to this information.** .. code-block:: ini - cp data_loading_config.yaml /path-to-your-dir/ + cp $METDATAIO_HOME/METdbLoad/test/Examples/example_load_specification.xml path-to-your-dir/load_specification.xml + +$METDATAIO is the path to the location of the cloned or forked METdataio source code. Replace the *path-to-your-dir* with the actual path to where this file is to be saved. -Change directory to the location where the *data_loading_config.yaml* file was copied. +Change directory to the location where the *example_load_specification.xml* file was copied. Make the necessary edits +to the required elements and delete any optional, unused/irrelevant elements. -Open the data_loading_config.yaml file: +.. dropdown:: The following is an explanation of the required and optional elements and any limitations -.. literalinclude:: ../../METdbLoad/sql/scripts/data_loading_config.yaml + *These are element names. The XML angle brackets (<>) as seen in the XML specification file are omitted* -Update the database information with information relevant to the database you are using: + **load_spec** + - **mandatory** + - top-level tag - * dbname + *The following elements pertain to logging into the database* + **connection** + - **mandatory** + - tag for connection information - * username + **management_system** + - **optional** + - indicates which database is in use + - recognized/expected values are one of the following: - * password + - aurora + - mysql + - mariadb - * host + - delete this element if not using - * port + **host** + - **mandatory** + - name of host/machine where database is installed + - format is *hostname*:*port number* + - minimum number of characters is 3 + - maximum number of characters is 67 + - allowable characters (combinations of any of these): + - upper and lower alphabetical characters (English) + - digits 0-9 + - ., -, _ (period, dash, underscore) -Update the path to the schema location, provide the full path to the *mv_sql_mysql.sql* schema file: + **database** + - **mandatory** + - name of the database + - maximum number of characters for database name is 124 + - allowable characters (combination of any of these): + - _,- (underscore, dash) + - upper and lower case alphabetical characters (English) + - digits 0-9 - * schema_location -Provide the name and full path to the *db_load_specification.xml* template file, this will be updated -with the settings in this YAML configuration to create a new XML specification file using these settings: + **user** + - **mandatory** + - user name + - minimum number of characters is 3 + - maximum number of characters is 32 + - allowable characters (combination of any of these): - * xml_specification + - upper and lower case alphabetical characters (English) + - digits 0-9 + - _,- (underscore, dash) -Provide the group and description. The databases in METviewer are grouped, provide the name of the appropriate -group and a brief description of the database in which the data is to be loaded: + **password** + - **mandatory** + - the password to access the database + - minimum number of characters is 3 + - maximum number of characters is 30 + - all characters are allowed - * group - * description + **local_infile** + - **optional** + - argument passed into 3rd party Python library pymysql -Provide the full path to the directory where the MET data to be loaded is saved: + - for establishing a connection to a MySQL server + - indicate whether the input file is local + - default is False + - enables use of the LOAD DATA LOCAL command - * data_dir + - Accepted value: + - Boolean value: True or False -Indicate which data types are to be loaded by setting the appropriate settings to True: + - True if loading local data + - False otherwise - * load_stat + - delete this element if loading of local data is not needed - * load_mode + - METdataio sets default to False if this element is absent - * load_mtd + *The following elements are used to define the format of multiple input data directories that are organized by datetime* + **date_list** + - **optional** + - only necessary when input data is organized based on datetime + - omit date_list entries if data resides in a singular directory + - multiple date_list elements are allowed - * load_mpr + - maximum number of date_lists is 5 + - differentiate different date_list definitions by the *name* attribute (i.e. name=) - * load_orank + **start** + - **mandatory** if date_list is being used + - start datetime -Generate the new XML specification file by running the following: -.. code-block:: ini + **end** + - **mandatory** if date_list is being used + - end datetime + + **inc** + - **mandatory** if date_list is being used + - increment/step size between start and end time + + - Example, if 6-hour increment: + - set inc to 0600 + - 0600 + + **format** + - **mandatory** if date_list is being used + - format of the datetime + + - Example, if 4 digit year month day hour: + - yyyyMMddHH + + *The following elements define various flags* + + **verbose** + - **mandatory** + - indicates the desired volume of output from the load module + + - TRUE resulting in more information + - FALSE resulting in less information + + **insert_size** + - **mandatory** + - An integer indicating the number of MET output file rows inserted with each INSERT statement + + - This value is most often 1 + + **stat_header_db_check** + - **optional** + - indicate whether a database query check for stat header information should be performed + - True or False (case insensitive) + + - **WARNING** enabling this feature (i.e. set to True) could significantly increase load time + + **mode_header_db_check** + - **optional** + - indicate whether a database query check for the MODE header information should be performed + - True or False (case insensitive) + + - **WARNING** enabling this feature (i.e. set to True) could significantly increase load time + + **mtd_header_db_check** + - **optional** + - indicate whether a database query check for the MODE TD header information should be performed + - True or False (case insensitive) + + - **WARNING** enabling this feature (i.e. set to True) could significantly increase load time + + + **drop_indexes** + - **optional** + - indicate whether to drop database indexes before loading new data + - True or False (case insensitive) + + **apply_indexes** + - **optional** + - indicate whether to apply database indexes + - True or False (case insensitive) + + **load_stat** + - **optional** + - indicate whether or not to load STAT data + - True or False (case insensitive) + + **load_mode** + - **optional** + - indicate whether or not to load MODE data + - True or False (case insensitive) + + **load_mtd** + - **optional** + - indicate whether or not to load MODE TD (MODE Time Domain) data + - True or False (case insensitive) + + **load_mpr** + - **optional** + - indicate whether or not to load MPR (matched pair) data + - True or False (case insensitive) + + **load_orank** + - **optional** + - indicate whether or not to load ORANK (observed rank) data + - True or False (case insensitive) + + **force_dup_file** + - **optional** + - indicate whether or not to force load paths/files that already exist + - True or False (case insensitive) + + *The following elements indicate which group the database should be assigned and a description* + **group** + - **optional** + - if undefined, the database will be placed under the NO GROUP group + - minimum number of characters is 1 + - maximum number of characters is 300 + - acceptable characters (English), any combination: + + - upper and/or lower case alphabetic characters + - any digits 0-9 + - _, . , - (underscore, period, dash) + + **description** + - **optional** + - description of the data in the database + - minimum number of characters is 1 + - maximum number of characters is 300 + - acceptable characters (English), any combination: + + - upper and/or lower case alphabetic characters + - any digits 0-9 + - _, . , - (underscore, period, dash) + + *The following defines the location of the input data to be loaded into the database* + + **folder_tmpl** + - **mandatory** + - only one folder template element is permitted - cd path-to-METdataio-source/METdataio/METdbLoad/sql/scripts + **field** - *Replace path-to-METdataio-source to the location where the METdataio source code is saved. - python generate_xml_spec.py path-to/data_loading_config.yaml - *Replace the path-to with the path to the directory you created to store the copy of the data_loading_config.yaml - file as specified earlier. -A new XML specification file *load_met.xml*, will be generated and saved in the -same directory where the YAML configuration file was copied. Load Data ========= @@ -128,7 +305,7 @@ The usage statement: INFO:root:--- *** --- Start METdbLoad --- *** --- - usage: met_db_load.py [-h] [-index] xmlfile [tmpdir [tmpdir ...]] + usage: python met_db_load.py [-h] [-index] xmlfile [tmpdir [tmpdir ...]] positional arguments: xmlfile Please provide required xml load_spec filename From 0d977a8bfa2d64a2ad44eb923e6ddd65d003df31 Mon Sep 17 00:00:00 2001 From: bikegeek Date: Wed, 23 Oct 2024 15:19:32 -0600 Subject: [PATCH 11/57] Add example for line_type element --- .../Examples/example_load_specification.xml | 26 ++++++++++++------- 1 file changed, 16 insertions(+), 10 deletions(-) diff --git a/METdbLoad/test/Examples/example_load_specification.xml b/METdbLoad/test/Examples/example_load_specification.xml index d67ee74f..4184e273 100644 --- a/METdbLoad/test/Examples/example_load_specification.xml +++ b/METdbLoad/test/Examples/example_load_specification.xml @@ -1,11 +1,11 @@ - your-host:### - your-database-name - your-dbusername - your-dbpassword + yourhost:1234 + your_database_name + your_db_username + your_password - + 2022050100 2022051200 @@ -18,7 +18,7 @@ 0600 yyyyMMddHH - + false 1 False @@ -30,8 +30,8 @@ false Regional Ensemble - /your-path-to/{config}/{fcst_init}/{mem}/metprd/{met_out}/ - + /var/autofs/mnt/hostmachine/projects/RRFS/prototype/met_out/{config}/{fcst_init}/{mem}/metprd/{met_out}/ + @@ -59,9 +59,15 @@ grid_stat_cmn point_stat_cmn + + CNT + ORANK + SEEPS_MPR + ME + - + true Load HREF and RTPS data for Spring 2022. - + From ff2d8184b8b7df6bcc9739228803e6209ae270eb Mon Sep 17 00:00:00 2001 From: bikegeek Date: Fri, 1 Nov 2024 12:09:49 -0600 Subject: [PATCH 12/57] Add checking for the load_files element to the schema --- METdbLoad/ush/load_specification_schema.xsd | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/METdbLoad/ush/load_specification_schema.xsd b/METdbLoad/ush/load_specification_schema.xsd index 00535268..6d5c5cf9 100644 --- a/METdbLoad/ush/load_specification_schema.xsd +++ b/METdbLoad/ush/load_specification_schema.xsd @@ -136,7 +136,16 @@ - + + + + + + + + + + From 4cea424d0a3806ac995f5714dc0924dbb3a51c0d Mon Sep 17 00:00:00 2001 From: bikegeek Date: Mon, 4 Nov 2024 09:46:43 -0700 Subject: [PATCH 13/57] Configs for loading specific files --- .../Examples/example_load_specific_files.xml | 78 +++++++++++++++++++ .../test/example_load_specific_files.xml | 78 +++++++++++++++++++ 2 files changed, 156 insertions(+) create mode 100644 METdbLoad/test/Examples/example_load_specific_files.xml create mode 100644 METdbLoad/test/example_load_specific_files.xml diff --git a/METdbLoad/test/Examples/example_load_specific_files.xml b/METdbLoad/test/Examples/example_load_specific_files.xml new file mode 100644 index 00000000..0924909f --- /dev/null +++ b/METdbLoad/test/Examples/example_load_specific_files.xml @@ -0,0 +1,78 @@ + + + yourhost:1234 + your_database_name + your_db_username + your_password + + + + 2022050100 + 2022051200 + 86400 + yyyyMMddHH + + + 2022050100 + 2022051200 + 0600 + yyyyMMddHH + + + false + 1 + False + false + FALSE + true + True + false + false + Regional Ensemble + + + /var/autofs/mnt/first_model.stat + /var/autofs/mnt/model_run1/file_1.stat + /var/autofs/mnt/model_run2/file_2.stat + + + + + + + + + + + + HREF_lag_offset + RTPS + + + mem01 + mem02 + mem03 + mem04 + mem05 + mem06 + mem07 + mem08 + mem09 + mem10 + + + grid_stat_cmn + point_stat_cmn + + + CNT + ORANK + SEEPS_MPR + ME + + + + true + Load HREF and RTPS data for Spring 2022. + + diff --git a/METdbLoad/test/example_load_specific_files.xml b/METdbLoad/test/example_load_specific_files.xml new file mode 100644 index 00000000..0924909f --- /dev/null +++ b/METdbLoad/test/example_load_specific_files.xml @@ -0,0 +1,78 @@ + + + yourhost:1234 + your_database_name + your_db_username + your_password + + + + 2022050100 + 2022051200 + 86400 + yyyyMMddHH + + + 2022050100 + 2022051200 + 0600 + yyyyMMddHH + + + false + 1 + False + false + FALSE + true + True + false + false + Regional Ensemble + + + /var/autofs/mnt/first_model.stat + /var/autofs/mnt/model_run1/file_1.stat + /var/autofs/mnt/model_run2/file_2.stat + + + + + + + + + + + + HREF_lag_offset + RTPS + + + mem01 + mem02 + mem03 + mem04 + mem05 + mem06 + mem07 + mem08 + mem09 + mem10 + + + grid_stat_cmn + point_stat_cmn + + + CNT + ORANK + SEEPS_MPR + ME + + + + true + Load HREF and RTPS data for Spring 2022. + + From 3c6ec73138cf530209fee5d2a65ce3eef1cbf9bc Mon Sep 17 00:00:00 2001 From: bikegeek Date: Mon, 4 Nov 2024 10:00:56 -0700 Subject: [PATCH 14/57] additional updates, more are pending. Mostly format changes. --- METdbLoad/conftest.py | 4 +- METdbLoad/docs/Users_Guide/load.rst | 10 +-- METdbLoad/test/test_xml.py | 6 +- docs/Users_Guide/load_data.rst | 112 +++++++++++++++------------- 4 files changed, 69 insertions(+), 63 deletions(-) diff --git a/METdbLoad/conftest.py b/METdbLoad/conftest.py index 81a42fe5..d5b6a06a 100644 --- a/METdbLoad/conftest.py +++ b/METdbLoad/conftest.py @@ -134,8 +134,8 @@ def mock_logger(): @pytest.fixture def get_specified_xml_loadfile( ) -> XmlLoadFile: """ - Retrieve the specified XML load specification filee. This is useful for using different XML - specification file for validating against recursive payloads, large payloads, etc. + Retrieve the specified XML load specification file. This is useful for using different XML + specification file for validating against recursive payloads, large payloads, etc. Args: xml_filename: The name of the XML file of interest diff --git a/METdbLoad/docs/Users_Guide/load.rst b/METdbLoad/docs/Users_Guide/load.rst index c73b21ba..61c7e698 100644 --- a/METdbLoad/docs/Users_Guide/load.rst +++ b/METdbLoad/docs/Users_Guide/load.rst @@ -79,18 +79,18 @@ increase load time. **:** **TRUE** or **FALSE**, this option indicates whether database indexes should be dropped prior to loading new data. -**:** **TRUE** or **FALSE**, this option indicates whether +**:** **TRUE** or **FALSE**, this option indicates whether database indexes should be created after loading new data. **:** The name of the group for the user interface. **:** A short description of the database. -| **:** A list structure containing individual MET output files to load into the database. -| -| **:** Follows the list of files after the previous tag, to end the list. +|**:** A list structure containing individual MET output files to load into the database. +| **:** Contains a single MET output file to load. +|**:** Follows the list of files after the previous tag, to end the list. -| **:** Contains a single MET output file to load. +| **:** A template string describing the file structure of the input MET files, which is populated with values specified in diff --git a/METdbLoad/test/test_xml.py b/METdbLoad/test/test_xml.py index 4857defc..eaa2143d 100644 --- a/METdbLoad/test/test_xml.py +++ b/METdbLoad/test/test_xml.py @@ -133,11 +133,11 @@ def test_tmp_xml(get_specified_xml_loadfile): f"schema {utils.LOAD_SPECIFICATION_SCHEMA}") pytest.fail(msg) -@pytest.mark.parametrize("xml_config", ["load_met_gha_prod.xml", "load_met_gha_new.xml"]) +@pytest.mark.parametrize("xml_config", ["load_met_gha_prod.xml", "load_met_gha_new.xml", "example_load_specific_files.xml"]) def test_db_xml(get_specified_xml_loadfile, xml_config): """ - Test validation against an XML specification file that is used in another test that compares two - databases. + Test validation against XML specification files used in another test that compares two + databases and an XML specification file that explicitly lists specific files to load. ValueError should be NOT be raised for the load_met_gha_prod/new specification files which are used on real data. """ diff --git a/docs/Users_Guide/load_data.rst b/docs/Users_Guide/load_data.rst index 89042e04..c06bb785 100644 --- a/docs/Users_Guide/load_data.rst +++ b/docs/Users_Guide/load_data.rst @@ -39,15 +39,16 @@ Create your own XML specification file by copying the example specification file *METdataio/METdbLoad/test/Examples/example_load_specification.xml* file to a location in your workspace. This file will contain the username and password to the database. -**Do not put this XML specification file where it can be read by anyone who should not have access to this information.** +**Do not save this XML specification file where it can be read by anyone who should not have access to this information.** .. code-block:: ini cp $METDATAIO_HOME/METdbLoad/test/Examples/example_load_specification.xml path-to-your-dir/load_specification.xml -$METDATAIO is the path to the location of the cloned or forked METdataio source code. -Replace the *path-to-your-dir* with the actual path to where this file is to be saved. +- $METDATAIO is the path to the location of the cloned or forked METdataio source code. + +- Replace the *path-to-your-dir* with the actual path to where this file is to be saved. Change directory to the location where the *example_load_specification.xml* file was copied. Make the necessary edits to the required elements and delete any optional, unused/irrelevant elements. @@ -56,58 +57,65 @@ to the required elements and delete any optional, unused/irrelevant elements. *These are element names. The XML angle brackets (<>) as seen in the XML specification file are omitted* - **load_spec** + + .. dropdown:: load_spec + - **mandatory** - - top-level tag + - top-level tag/element + - container for other elements that define connection information, flags, data input, etc. *The following elements pertain to logging into the database* - **connection** - - **mandatory** - - tag for connection information - - **management_system** - - **optional** - - indicates which database is in use - - recognized/expected values are one of the following: - - - aurora - - mysql - - mariadb - - - delete this element if not using - - **host** - - **mandatory** - - name of host/machine where database is installed - - format is *hostname*:*port number* - - minimum number of characters is 3 - - maximum number of characters is 67 - - allowable characters (combinations of any of these): - - - upper and lower alphabetical characters (English) - - digits 0-9 - - ., -, _ (period, dash, underscore) - - **database** - - **mandatory** - - name of the database - - maximum number of characters for database name is 124 - - allowable characters (combination of any of these): - - _,- (underscore, dash) - - upper and lower case alphabetical characters (English) - - digits 0-9 + .. dropdown:: connection + - **mandatory** + - tag for connection information - **user** - - **mandatory** - - user name - - minimum number of characters is 3 - - maximum number of characters is 32 - - allowable characters (combination of any of these): + .. dropdown:: management_system + + - **optional** + - indicates which database is in use + - recognized/expected values are one of the following: + + - aurora + - mysql + - mariadb + + - delete this element if not using + + .. dropdown:: host - - upper and lower case alphabetical characters (English) - - digits 0-9 - - _,- (underscore, dash) + - **mandatory** + - name of host/machine where database is installed + - format is *hostname*:*port number* + - minimum number of characters is 3 + - maximum number of characters is 67 + - allowable characters (combinations of any of these): + + - upper and lower alphabetical characters (English) + - digits 0-9 + - ., -, _ (period, dash, underscore) + + .. dropdown :: database + + - **mandatory** + - name of the database + - maximum number of characters for database name is 124 + - allowable characters (combination of any of these): + - _,- (underscore, dash) + - upper and lower case alphabetical characters (English) + - digits 0-9 + + .. dropdown:: user + + - **mandatory** + - user name + - minimum number of characters is 3 + - maximum number of characters is 32 + - allowable characters (combination of any of these): + + - upper and lower case alphabetical characters (English) + - digits 0-9 + - _,- (underscore, dash) **password** - **mandatory** @@ -116,7 +124,6 @@ to the required elements and delete any optional, unused/irrelevant elements. - maximum number of characters is 30 - all characters are allowed - **local_infile** - **optional** - argument passed into 3rd party Python library pymysql @@ -150,7 +157,6 @@ to the required elements and delete any optional, unused/irrelevant elements. - **mandatory** if date_list is being used - start datetime - **end** - **mandatory** if date_list is being used - end datetime @@ -206,7 +212,6 @@ to the required elements and delete any optional, unused/irrelevant elements. - **WARNING** enabling this feature (i.e. set to True) could significantly increase load time - **drop_indexes** - **optional** - indicate whether to drop database indexes before loading new data @@ -250,6 +255,7 @@ to the required elements and delete any optional, unused/irrelevant elements. *The following elements indicate which group the database should be assigned and a description* **group** - **optional** + - the name of the database group (databases are grouped in METviewer: e.g. Test, - if undefined, the database will be placed under the NO GROUP group - minimum number of characters is 1 - maximum number of characters is 300 @@ -261,7 +267,7 @@ to the required elements and delete any optional, unused/irrelevant elements. **description** - **optional** - - description of the data in the database + - description of the database - minimum number of characters is 1 - maximum number of characters is 300 - acceptable characters (English), any combination: From c70cbae959af6e6a195375c899ff8be247b249db Mon Sep 17 00:00:00 2001 From: bikegeek Date: Mon, 4 Nov 2024 12:00:40 -0700 Subject: [PATCH 15/57] Removed unused content for folder_tmpl information. --- .../test/example_load_specific_files.xml | 49 +------------------ 1 file changed, 1 insertion(+), 48 deletions(-) diff --git a/METdbLoad/test/example_load_specific_files.xml b/METdbLoad/test/example_load_specific_files.xml index 0924909f..a99ef2cd 100644 --- a/METdbLoad/test/example_load_specific_files.xml +++ b/METdbLoad/test/example_load_specific_files.xml @@ -6,19 +6,6 @@ your_password - - 2022050100 - 2022051200 - 86400 - yyyyMMddHH - - - 2022050100 - 2022051200 - 0600 - yyyyMMddHH - - false 1 False @@ -37,41 +24,7 @@ - - - - - - - - - HREF_lag_offset - RTPS - - - mem01 - mem02 - mem03 - mem04 - mem05 - mem06 - mem07 - mem08 - mem09 - mem10 - - - grid_stat_cmn - point_stat_cmn - - - CNT - ORANK - SEEPS_MPR - ME - - - + true Load HREF and RTPS data for Spring 2022. From 8fb4b1727544e7e77b3a2b0e1b2b3730fb169bc6 Mon Sep 17 00:00:00 2001 From: bikegeek Date: Mon, 4 Nov 2024 12:05:47 -0700 Subject: [PATCH 16/57] folder_tmpl min occurrence changed to 0 because of load_files support --- METdbLoad/ush/load_specification_schema.xsd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/METdbLoad/ush/load_specification_schema.xsd b/METdbLoad/ush/load_specification_schema.xsd index 6d5c5cf9..08eac645 100644 --- a/METdbLoad/ush/load_specification_schema.xsd +++ b/METdbLoad/ush/load_specification_schema.xsd @@ -147,7 +147,7 @@ - + From 54a3cd00e606a052be24d74ac65b7116bbd31a3e Mon Sep 17 00:00:00 2001 From: bikegeek Date: Mon, 4 Nov 2024 15:21:13 -0700 Subject: [PATCH 17/57] Accidentally removed the element. Put this back in. --- .../Examples/example_load_specific_files.xml | 59 ++----------------- .../test/example_load_specific_files.xml | 14 ++--- 2 files changed, 11 insertions(+), 62 deletions(-) diff --git a/METdbLoad/test/Examples/example_load_specific_files.xml b/METdbLoad/test/Examples/example_load_specific_files.xml index 0924909f..d80fad59 100644 --- a/METdbLoad/test/Examples/example_load_specific_files.xml +++ b/METdbLoad/test/Examples/example_load_specific_files.xml @@ -5,21 +5,8 @@ your_db_username your_password - - - 2022050100 - 2022051200 - 86400 - yyyyMMddHH - - - 2022050100 - 2022051200 - 0600 - yyyyMMddHH - - - false + + true 1 False false @@ -31,48 +18,12 @@ Regional Ensemble - /var/autofs/mnt/first_model.stat - /var/autofs/mnt/model_run1/file_1.stat - /var/autofs/mnt/model_run2/file_2.stat + /met_out/mode/mode_MASK_POLY_300000L_20120410_180000V_060000A_cts.txt + /met_out/rhist/ensemble_stat_RRFS_GEFS_GF.SPP.SPPT_RETOP_MRMS_20220507_120000V.stat - - - - - - - - - - HREF_lag_offset - RTPS - - - mem01 - mem02 - mem03 - mem04 - mem05 - mem06 - mem07 - mem08 - mem09 - mem10 - - - grid_stat_cmn - point_stat_cmn - - - CNT - ORANK - SEEPS_MPR - ME - - true - Load HREF and RTPS data for Spring 2022. + Load MODE and RHIST for load_files. diff --git a/METdbLoad/test/example_load_specific_files.xml b/METdbLoad/test/example_load_specific_files.xml index a99ef2cd..d80fad59 100644 --- a/METdbLoad/test/example_load_specific_files.xml +++ b/METdbLoad/test/example_load_specific_files.xml @@ -5,8 +5,8 @@ your_db_username your_password - - false + + true 1 False false @@ -18,14 +18,12 @@ Regional Ensemble - /var/autofs/mnt/first_model.stat - /var/autofs/mnt/model_run1/file_1.stat - /var/autofs/mnt/model_run2/file_2.stat + /met_out/mode/mode_MASK_POLY_300000L_20120410_180000V_060000A_cts.txt + /met_out/rhist/ensemble_stat_RRFS_GEFS_GF.SPP.SPPT_RETOP_MRMS_20220507_120000V.stat - - + true - Load HREF and RTPS data for Spring 2022. + Load MODE and RHIST for load_files. From 94af151e62d120b172e42277cf44fa3671736220 Mon Sep 17 00:00:00 2001 From: bikegeek Date: Mon, 4 Nov 2024 17:57:28 -0700 Subject: [PATCH 18/57] Added description of flags. --- docs/Users_Guide/load_data.rst | 220 ++++++++++++++++++--------------- 1 file changed, 120 insertions(+), 100 deletions(-) diff --git a/docs/Users_Guide/load_data.rst b/docs/Users_Guide/load_data.rst index c06bb785..e5562b0b 100644 --- a/docs/Users_Guide/load_data.rst +++ b/docs/Users_Guide/load_data.rst @@ -55,7 +55,7 @@ to the required elements and delete any optional, unused/irrelevant elements. .. dropdown:: The following is an explanation of the required and optional elements and any limitations - *These are element names. The XML angle brackets (<>) as seen in the XML specification file are omitted* + *These are element names. The XML angle brackets (<>) as seen in the XML specification file are omitted for simplicity* .. dropdown:: load_spec @@ -70,184 +70,204 @@ to the required elements and delete any optional, unused/irrelevant elements. - **mandatory** - tag for connection information - .. dropdown:: management_system + .. dropdown:: management_system - - **optional** - - indicates which database is in use - - recognized/expected values are one of the following: + - **optional** + - indicates which database is in use + - recognized/expected values are one of the following: + + - aurora + - mysql + - mariadb + + - delete this element if not using - - aurora - - mysql - - mariadb + .. dropdown:: host - - delete this element if not using + - **mandatory** + - name of host/machine where database is installed + - format is *hostname*:*port number* + - minimum number of characters is 3 + - maximum number of characters is 67 + - allowable characters (combinations of any of these): - .. dropdown:: host + - upper and lower alphabetical characters (English) + - digits 0-9 + - ., -, _ (period, dash, underscore) + + .. dropdown:: database - **mandatory** - - name of host/machine where database is installed - - format is *hostname*:*port number* + - name of the database + - maximum number of characters for database name is 124 + - allowable characters (combination of any of these): + - _,- (underscore, dash) + - upper and lower case alphabetical characters (English) + - digits 0-9 + + .. dropdown:: user + + - **mandatory** + - user name - minimum number of characters is 3 - - maximum number of characters is 67 - - allowable characters (combinations of any of these): + - maximum number of characters is 32 + - allowable characters (combination of any of these): - - upper and lower alphabetical characters (English) - - digits 0-9 - - ., -, _ (period, dash, underscore) + - upper and lower case alphabetical characters (English) + - digits 0-9 + - _,- (underscore, dash) - .. dropdown :: database + .. dropdown:: password - - **mandatory** - - name of the database - - maximum number of characters for database name is 124 - - allowable characters (combination of any of these): - - _,- (underscore, dash) - - upper and lower case alphabetical characters (English) - - digits 0-9 + - **mandatory** + - the password to access the database + - minimum number of characters is 3 + - maximum number of characters is 30 + - all characters are allowed - .. dropdown:: user + .. dropdown:: local_infile - - **mandatory** - - user name - - minimum number of characters is 3 - - maximum number of characters is 32 - - allowable characters (combination of any of these): + - **optional** + - argument passed into 3rd party Python library pymysql - - upper and lower case alphabetical characters (English) - - digits 0-9 - - _,- (underscore, dash) + - for establishing a connection to a MySQL server + - indicate whether the input file is local + - default is False + - enables use of the LOAD DATA LOCAL command - **password** - - **mandatory** - - the password to access the database - - minimum number of characters is 3 - - maximum number of characters is 30 - - all characters are allowed + - Accepted value: + - Boolean value: True or False + + - True if loading local data + - False otherwise - **local_infile** - - **optional** - - argument passed into 3rd party Python library pymysql + - delete this element if loading of local data is not needed - - for establishing a connection to a MySQL server - - indicate whether the input file is local - - default is False - - enables use of the LOAD DATA LOCAL command + - METdataio sets default to False if this element is absent - - Accepted value: - - Boolean value: True or False + *The following elements are used to define the format of multiple input data directories that are (optionally) organized by datetime* + .. dropdown:: date_list - - True if loading local data - - False otherwise + - **optional** + - only necessary when input data is organized based on datetime + - omit date_list entries if data resides in a singular directory + - multiple date_list elements are allowed - - delete this element if loading of local data is not needed + - maximum number of date_lists is 5 + - differentiate different date_list definitions by the *name* attribute (i.e. name=) - - METdataio sets default to False if this element is absent + .. dropdown:: start - *The following elements are used to define the format of multiple input data directories that are organized by datetime* - **date_list** - - **optional** - - only necessary when input data is organized based on datetime - - omit date_list entries if data resides in a singular directory - - multiple date_list elements are allowed + - **mandatory** if date_list is being used + - start datetime - - maximum number of date_lists is 5 - - differentiate different date_list definitions by the *name* attribute (i.e. name=) + .. dropdown:: end - **start** - - **mandatory** if date_list is being used - - start datetime + - **mandatory** if date_list is being used + - end datetime - **end** - - **mandatory** if date_list is being used - - end datetime + .. dropdown:: inc - **inc** - - **mandatory** if date_list is being used - - increment/step size between start and end time + - **mandatory** if date_list is being used + - increment/step size between start and end time - - Example, if 6-hour increment: - - set inc to 0600 - - 0600 + - Example, if 6-hour increment: + - set inc to 0600 + - 0600 - **format** - - **mandatory** if date_list is being used - - format of the datetime + .. dropdown:: format - - Example, if 4 digit year month day hour: - - yyyyMMddHH + - **mandatory** if date_list is being used + - format of the datetime + + - Example, if 4 digit year month day hour: + - yyyyMMddHH *The following elements define various flags* - **verbose** - - **mandatory** - - indicates the desired volume of output from the load module + .. dropdown:: verbose + + - **mandatory** + - indicates the desired volume of output from the load module - TRUE resulting in more information - FALSE resulting in less information - **insert_size** - - **mandatory** - - An integer indicating the number of MET output file rows inserted with each INSERT statement + .. dropdown:: insert_size - - This value is most often 1 + - **mandatory** + - An integer indicating the number of MET output file rows inserted with each INSERT statement - **stat_header_db_check** - - **optional** - - indicate whether a database query check for stat header information should be performed - - True or False (case insensitive) + - This value is most often 1 - - **WARNING** enabling this feature (i.e. set to True) could significantly increase load time + .. dropdown:: stat_header_db_check + + - **optional** + - indicate whether a database query check for stat header information should be performed + - True or False (case insensitive) + + - **WARNING** enabling this feature (i.e. set to True) could significantly increase load time + + .. dropdown:: mode_header_db_check - **mode_header_db_check** - **optional** - indicate whether a database query check for the MODE header information should be performed - True or False (case insensitive) - **WARNING** enabling this feature (i.e. set to True) could significantly increase load time - **mtd_header_db_check** + .. dropdown:: **mtd_header_db_check** + - **optional** - indicate whether a database query check for the MODE TD header information should be performed - True or False (case insensitive) - **WARNING** enabling this feature (i.e. set to True) could significantly increase load time - **drop_indexes** + .. dropdown:: drop_indexes + - **optional** - indicate whether to drop database indexes before loading new data - True or False (case insensitive) - **apply_indexes** + .. dropdown:: apply_indexes + - **optional** - indicate whether to apply database indexes - True or False (case insensitive) - **load_stat** + .. dropdown:: load_stat + - **optional** - indicate whether or not to load STAT data - True or False (case insensitive) - **load_mode** + .. dropdown:: load_mode + - **optional** - indicate whether or not to load MODE data - True or False (case insensitive) - **load_mtd** + .. dropdown:: load_mtd + - **optional** - indicate whether or not to load MODE TD (MODE Time Domain) data - True or False (case insensitive) - **load_mpr** + .. dropdown:: load_mpr + - **optional** - indicate whether or not to load MPR (matched pair) data - True or False (case insensitive) - **load_orank** + .. dropdown:: load_orank + - **optional** - indicate whether or not to load ORANK (observed rank) data - True or False (case insensitive) - **force_dup_file** + .. dropdown:: force_dup_file + - **optional** - indicate whether or not to force load paths/files that already exist - True or False (case insensitive) From e3e07086cffd9def34f14ebef744d6a95b855d07 Mon Sep 17 00:00:00 2001 From: bikegeek Date: Tue, 5 Nov 2024 13:00:03 -0700 Subject: [PATCH 19/57] add the valid_times in the folder_tmpl to be consistent with the load_val elements in the exmample --- METdbLoad/test/Examples/example_load_specification.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/METdbLoad/test/Examples/example_load_specification.xml b/METdbLoad/test/Examples/example_load_specification.xml index 4184e273..e39c4ea2 100644 --- a/METdbLoad/test/Examples/example_load_specification.xml +++ b/METdbLoad/test/Examples/example_load_specification.xml @@ -30,7 +30,7 @@ false Regional Ensemble - /var/autofs/mnt/hostmachine/projects/RRFS/prototype/met_out/{config}/{fcst_init}/{mem}/metprd/{met_out}/ + /var/autofs/mnt/hostmachine/projects/RRFS/prototype/met_out/{config}/{fcst_init}/{mem}/{valid_times}/metprd/{met_out}/ From 7dc773a40fb4f0f1ebef04c94179a76375df1668 Mon Sep 17 00:00:00 2001 From: bikegeek Date: Tue, 5 Nov 2024 15:32:00 -0700 Subject: [PATCH 20/57] Changed order of elements for consistency --- .../test/Examples/example_load_specification.xml | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/METdbLoad/test/Examples/example_load_specification.xml b/METdbLoad/test/Examples/example_load_specification.xml index e39c4ea2..8a426b73 100644 --- a/METdbLoad/test/Examples/example_load_specification.xml +++ b/METdbLoad/test/Examples/example_load_specification.xml @@ -33,17 +33,14 @@ /var/autofs/mnt/hostmachine/projects/RRFS/prototype/met_out/{config}/{fcst_init}/{mem}/{valid_times}/metprd/{met_out}/ - - - - - - HREF_lag_offset RTPS - + + + + mem01 mem02 mem03 @@ -55,6 +52,9 @@ mem09 mem10 + + + grid_stat_cmn point_stat_cmn From 23f708bc80860ca853e5d231a7823e8d0ec09d87 Mon Sep 17 00:00:00 2001 From: bikegeek Date: Tue, 5 Nov 2024 16:54:00 -0700 Subject: [PATCH 21/57] Added content for the load_val/field for config --- docs/Users_Guide/load_data.rst | 340 +++++++++++++++++---------------- 1 file changed, 176 insertions(+), 164 deletions(-) diff --git a/docs/Users_Guide/load_data.rst b/docs/Users_Guide/load_data.rst index e5562b0b..b7744b3e 100644 --- a/docs/Users_Guide/load_data.rst +++ b/docs/Users_Guide/load_data.rst @@ -20,8 +20,16 @@ XML schema and conform to size and number of element limitations. .. dropdown:: This is an **example XML specification file** that is **valid**: + For defining data organized by dates using the folder_tmpl element: + .. literalinclude:: ../../METdbLoad/test/Examples/example_load_specification.xml +.. dropdown:: This is another **example XML specification file** that is **valid**: + + For specifying a list of input data files using the load_files element: + + .. literalinclude:: ../../METdbLoad/test/Examples/example_load_specific_files.xml + .. dropdown:: This is the **XML schema** that is used to **validate the XML specification file**: @@ -57,6 +65,10 @@ to the required elements and delete any optional, unused/irrelevant elements. *These are element names. The XML angle brackets (<>) as seen in the XML specification file are omitted for simplicity* + **!!!NOTE!!!** + + The **order of the elements** in the XML specification file is crucial. **DO NOT** modify the order of the following elements. + Indentation is used to indicate hierarchical relationships between elements. .. dropdown:: load_spec @@ -65,6 +77,7 @@ to the required elements and delete any optional, unused/irrelevant elements. - container for other elements that define connection information, flags, data input, etc. *The following elements pertain to logging into the database* + .. dropdown:: connection - **mandatory** @@ -146,16 +159,35 @@ to the required elements and delete any optional, unused/irrelevant elements. - METdataio sets default to False if this element is absent *The following elements are used to define the format of multiple input data directories that are (optionally) organized by datetime* + .. dropdown:: date_list - **optional** - - only necessary when input data is organized based on datetime + - for describing data organized in datetime subdirectories - omit date_list entries if data resides in a singular directory - multiple date_list elements are allowed - maximum number of date_lists is 5 - differentiate different date_list definitions by the *name* attribute (i.e. name=) + **Example**: + + /var/autofs/mnt/hostmachine/projects/RRFS/prototype/met_out/{config}/{fcst_init}/{mem}/{valid_times}/metprd/{met_out} + + - the *fcst_init* and *valid_times* subdirectories are based on datetime + - assign the fcst_init subdirectory to a descriptively named date_list attribute: + + e.g. + + - this attribute name will be used in the load_val element within the folder_tmpl element to describe the + {fcst_init} subdirectory template + + - assign the valid_times subdirectory to a descriptively named date_list attribute: + + e.g. + + - this attribute name will be used in the load_val element within the folder_tmpl element to describe the {valid_times} subdirectory template + .. dropdown:: start - **mandatory** if date_list is being used @@ -180,7 +212,7 @@ to the required elements and delete any optional, unused/irrelevant elements. - **mandatory** if date_list is being used - format of the datetime - - Example, if 4 digit year month day hour: + - For example, to specify 4 digit year, 2 digit month, 2 digit day, and 2 digit hour: - yyyyMMddHH *The following elements define various flags* @@ -216,7 +248,7 @@ to the required elements and delete any optional, unused/irrelevant elements. - **WARNING** enabling this feature (i.e. set to True) could significantly increase load time - .. dropdown:: **mtd_header_db_check** + .. dropdown:: mtd_header_db_check - **optional** - indicate whether a database query check for the MODE TD header information should be performed @@ -273,9 +305,11 @@ to the required elements and delete any optional, unused/irrelevant elements. - True or False (case insensitive) *The following elements indicate which group the database should be assigned and a description* - **group** + + .. dropdown:: group + - **optional** - - the name of the database group (databases are grouped in METviewer: e.g. Test, + - the name of the database group (databases are grouped in METviewer: e.g. Testing) - if undefined, the database will be placed under the NO GROUP group - minimum number of characters is 1 - maximum number of characters is 300 @@ -285,7 +319,8 @@ to the required elements and delete any optional, unused/irrelevant elements. - any digits 0-9 - _, . , - (underscore, period, dash) - **description** + .. dropdown:: description + - **optional** - description of the database - minimum number of characters is 1 @@ -296,15 +331,140 @@ to the required elements and delete any optional, unused/irrelevant elements. - any digits 0-9 - _, . , - (underscore, period, dash) - *The following defines the location of the input data to be loaded into the database* + *The following defines the location of the input data to be loaded into the database based on data organized by datetime (and any other criteria)* - **folder_tmpl** - - **mandatory** - - only one folder template element is permitted + .. dropdown:: folder_tmpl + + - **mandatory** only if data is organized in directories that can be described by templates + - only one folder template element is permitted (i.e. only one ... ) + - **NOTE** the *date_list* element **MUST BE DEFINED** (see above in the *date_list* description) if any subdirectories are based on datetime + + *Specify the directory where the data is located in one of the following methods:* + + .. dropdown:: Using value templates for directories: + + Example: + + **/var/autofs/mnt/hostmachine/projects/RRFS/prototype/met_out/{config}/{fcst_init}/{mem}/{valid_times}/metprd/{met_out}** + + - data is organized into various directories based on datetime, and other criteria + + - use { } around "variable" names (in XML, these indicate attribute value templates) + + **config**, **fcst_init**, **mem**, **valid_times**, and **met_out** are attribute value template values that *must* be defined under the load_val element (for more details, refer to the *load_val* description below) + + .. dropdown:: Specify a single directory where all data reside: + + Example: + + **/var/autofs/mnt/hostmachine/projects/RRFS/prototype/met_out/mem00/metprd/all_runs** + + - **all** datafiles are located under this directory (indicate the full path) + + .. dropdown:: load_val + + - **optional** if *folder_tmpl* specifies a single directory where all data resides + + - **mandatory** if folder_tmpl has datetime subdirectories + - *field* sub-elements correspond to each attribute value template (i.e. variable names enclosed in {}) + + + .. dropdown:: field + + - **mandatory** if *folder_tmpl* has subdirectories that are datetimes + - each *field* element defines the attribute value template in the directory structure (i.e. the variable inside the {}) + - *field* elements can have one or more *val* sub-elements that can specify more subdirectories + - *field* elements can have one or more *date_list* sub-elements for subdirectories that are datetimes + + *For this folder_tmpl example:* + + **/var/autofs/mnt/hostmachine/projects/RRFS/prototype/met_out/{config}/{fcst_init}/{mem}/{valid_times}/metprd/{met_out}** + + *The following are the name attributes for the field* element for the above example: + + .. dropdown:: config + + - corresponds to the {config} template: + + + + .. dropdown:: val + + - for defining non-datetime subdirectories + + - maximum number of *val* elements is 100 + + e.g.: + + + HREF_lag_offset + + RTPS + + + + .. dropdown:: fcst_init + + - corresponds to the {fcst_init} template: + + - **field** + .. dropdown:: date_list + - the *name* attribute corresponds to one of the *date_list* attribute names + - in this case, this corresponds to the *folder_dates* attribute name: + + + + 2022050100 + 2022051200 + 86400 + yyyyMMddHH + + + + + if *fcst_init* subdirectory is based on the *date_list* named *folder_dates* + + then the following is expected + + + + + + + + .. dropdown:: mem + + - corresponds to the {mem} template: + + + + - *val* element to specify subdirectories that are NOT datetimes + + - maximum number of vals: 100 + + .. dropdown:: valid_times + + - since this is a datetime subdirectory, a *date_list* element is expected + - + - the "valid_dates" name attribute value matches what is defined in the *date_list* element at the top of the XML specification file + + .. dropdown:: met_out + + .. dropdown:: val + + - one or more *val* elements + - define any other subdirectories that are NOT datetimes + + + + *The following defines the location of specific input data files to be loaded into the database* + + .. dropdown:: load_files + + - **mandatory** if specifying individual data file(s) @@ -347,58 +507,7 @@ tag is and it contains the following elements, divided into functional sections: - * **:** **TRUE** or **FALSE**, this option indicates whether or - not to load STAT data. - - * **:** **TRUE** or **FALSE**, this option indicates whether or - not to load MODE data. - - * **:** **TRUE** or **FALSE**, this option indicates whether or - not to load MODE TD data. - - * **:** **TRUE** or **FALSE**, this option indicates whether or not - to load matched pair data. - * **:** **TRUE** or **FALSE**, this option indicates whether or - not to load observed rank data. - - * **:** **TRUE** or **FALSE**, this option indicates whether - or not to force load paths/files that are already present. - - * **:** **TRUE** or **FALSE**, this option indicates the desired - volume of output from the load module, with TRUE resulting in more - information and FALSE resulting in less information. - - * **:** An integer indicating the number of MET output file rows - that are inserted with each INSERT statement. This value is most often 1. - - * **:** **TRUE** or **FALSE**, this option indicates - whether a database query check for stat header information should be - performed - **WARNING:** enabling this feature could significantly - increase load time. - - **NOTE:** **** has been removed; remove it - from the XML load specification document. - - * **:** **TRUE** or **FALSE**, this option indicates - whether a database query check for MODE header information should be - performed - **WARNING:** enabling this feature could significantly - increase load time. - - * **:** **TRUE** or **FALSE**, this option indicates - whether a database query check for MODE TD header information should - be performed - **WARNING:** enabling this feature could significantly - increase load time. - - * **:** **TRUE** or **FALSE**, this option indicates whether - database indexes should be dropped prior to loading new data. - - * **:** **TRUE** or **FALSE**, this option indicates whether - database indexes should be created after loading new data. - - * **:** The name of the group for the user interface. - - * **:** A short description of the database. * **:** A list structure containing individual MET output files to load into the database. @@ -408,28 +517,17 @@ functional sections: * **:** Contains a single MET output file to load. - * **:** A template string describing the file structure of - the input MET files, which is populated with values specified in - the **** tag structure. - * **:** A tree structure containing values used to populate - the **** template. - * **:** A template value, its name is specified by the attribute - name, and its values are specified by its children **** tags. + * **:** A template value, its name is specified by the attribute name, and its values are specified by its children **** tags. - * **:** A single template value which will slot into the template - in the value specified by the parent field's name. + * **:** A single template value which will slot into the template in the value specified by the parent field's name. - * **:** Specifies a previously declared **** - element, using the name attribute, which represents a list of dates - in a particular format. + * **:** Specifies a previously declared **** element, using the name attribute, which represents a list of dates in a particular format. - * **:** A list structure containing the MET output file line - types to load. If omitted, all line types are loaded. + * **:** A list structure containing the MET output file line types to load. If omitted, all line types are loaded. - * **:** Contains a single MET output file line type to be loaded, - for example, CNT. + * **:** Contains a single MET output file line type to be loaded, for example, CNT. * **:** If present, creates a record in the instance_info database database table with a note containing the body of this tag @@ -438,100 +536,14 @@ functional sections: not to save the load xml; only effective if **** is present - default: TRUE - **Note** - If is used, at least one entry should be present. - For example, if the path is: - - .. code-block:: XML - - /path/to/data - change it to - .. code-block:: XML - /path/to/{type} - - - data - - -Additional Loading Options --------------------------- -The load_met.xml specification file created above loads the entire dataset specified in the data_dir setting in the -YAML config file, data_loading_config.yaml. -A subset of the data can be selected by date and field names (i.e. by model, valid_time, vx_mask, etc.). -The load_met.xml specification file can be further modified to accomplish this by adding the date_list and -field_name elements to the XML specification file. - -Here is a simple example: - -.. code-block:: XML - - - - kemosabe:3306 - mv_db_hwt - pgoldenb - pgoldenb - - - - 2010051914 - 2010051915 - 3600 - yyyyMMddHH - - - - false - 1 - true - false - true - Group name - true - true - true - false - - /d1/data/{model}/{vx_mask}/{valid_time} - - - arw - nmm - - - - - - - - FULL - SWC - - - - - -In this example, the load module would attempt to load any files with the -suffix .stat in the following folders. - -.. code-block:: ini - /d1/data/arw/FULL/2010051914 - /d1/data/arw/SWC/2010051914 - /d1/data/nmm/FULL/2010051914 - /d1/data/nmm/SWC/2010051914 - /d1/data/arw/FULL/2010051915 - /d1/data/arw/SWC/2010051915 - /d1/data/nmm/FULL/2010051915 - /d1/data/nmm/SWC/2010051915 - ... Troubleshooting --------------- From 425464e3d9c8db984e442a080f5e282af85de242 Mon Sep 17 00:00:00 2001 From: bikegeek Date: Wed, 6 Nov 2024 07:28:20 -0700 Subject: [PATCH 22/57] Finished content for folder_tmpl --- docs/Users_Guide/load_data.rst | 130 +++++++++++++++++++++++++++------ 1 file changed, 106 insertions(+), 24 deletions(-) diff --git a/docs/Users_Guide/load_data.rst b/docs/Users_Guide/load_data.rst index b7744b3e..26d9629f 100644 --- a/docs/Users_Guide/load_data.rst +++ b/docs/Users_Guide/load_data.rst @@ -335,9 +335,9 @@ to the required elements and delete any optional, unused/irrelevant elements. .. dropdown:: folder_tmpl - - **mandatory** only if data is organized in directories that can be described by templates + - **mandatory** only if data is organized in directories that are described by templates or if all data resides under one directory - only one folder template element is permitted (i.e. only one ... ) - - **NOTE** the *date_list* element **MUST BE DEFINED** (see above in the *date_list* description) if any subdirectories are based on datetime + - **NOTE** the *date_list* element **MUST BE DEFINED** (see above in the *date_list* description) if **any subdirectories are based on datetime** *Specify the directory where the data is located in one of the following methods:* @@ -366,19 +366,28 @@ to the required elements and delete any optional, unused/irrelevant elements. - **optional** if *folder_tmpl* specifies a single directory where all data resides - **mandatory** if folder_tmpl has datetime subdirectories - - *field* sub-elements correspond to each attribute value template (i.e. variable names enclosed in {}) + - *field* elements correspond to each attribute value template (i.e. variable names enclosed in {}) .. dropdown:: field - **mandatory** if *folder_tmpl* has subdirectories that are datetimes - each *field* element defines the attribute value template in the directory structure (i.e. the variable inside the {}) - - *field* elements can have one or more *val* sub-elements that can specify more subdirectories - - *field* elements can have one or more *date_list* sub-elements for subdirectories that are datetimes + - *val* elements that can specify more subdirectories - *For this folder_tmpl example:* + - optional + - necessary when specifying subdirectories that are not datetimes (e.g. /path-to/model_A/air_quality/) + - maximum number of val elements: 100 - **/var/autofs/mnt/hostmachine/projects/RRFS/prototype/met_out/{config}/{fcst_init}/{mem}/{valid_times}/metprd/{met_out}** + - *date_list* elements for subdirectories that are datetimes + + - optional + - mandatory when subdirectories are datetimes (e.g. /path-to/20240101/model/) + - maximum number of date_list elements: 5 + + *For this folder_tmpl example, which has datetime subdirectories:* + + .. dropdown:: /var/autofs/mnt/hostmachine/projects/RRFS/prototype/met_out/{config}/{fcst_init}/{mem}/{valid_times}/metprd/{met_out} *The following are the name attributes for the field* element for the above example: @@ -413,50 +422,123 @@ to the required elements and delete any optional, unused/irrelevant elements. - the *name* attribute corresponds to one of the *date_list* attribute names - - in this case, this corresponds to the *folder_dates* attribute name: + + + **** - + + + .. dropdown:: The "folder_dates" name attribute was assigned in the date_list portion of the XML specification file + + **** 2022050100 + 2022051200 + 86400 + yyyyMMddHH - if *fcst_init* subdirectory is based on the *date_list* named *folder_dates* + .. dropdown:: mem - then the following is expected + - corresponds to the {mem} template: - + - + .. dropdown:: val - + - for defining non-datetime subdirectories - .. dropdown:: mem + - maximum number of *val* elements is 100 - - corresponds to the {mem} template: + e.g.: - + + + mem01 + + mem02 + + mem03 + + mem04 + + mem05 + + mem06 + + mem07 + + mem08 + + mem09 + + mem10 - - *val* element to specify subdirectories that are NOT datetimes + - - maximum number of vals: 100 .. dropdown:: valid_times - - since this is a datetime subdirectory, a *date_list* element is expected - - - - the "valid_dates" name attribute value matches what is defined in the *date_list* element at the top of the XML specification file + - corresponds to the {valid_times} template: + + + + .. dropdown:: valid_times + + - the *name* attribute corresponds to one of the *date_list* attribute names + + + + **** + + + + .. dropdown:: the "valid_dates" name attribute value was assigned in the date_list portion of the XML specification file + + **** + + 2022050100 + + 2022051200 + + 0600 + + yyyyMMddHH + + .. dropdown:: met_out + - corresponds to the {met_out} template + + + .. dropdown:: val - - one or more *val* elements - - define any other subdirectories that are NOT datetimes + - for defining non-datetime subdirectories + + - maximum number of *val* elements is 100 + + e.g.: + + + + grid_stat_cmn + + point_stat_cmn + + + + *For this folder_tmpl example, which has all the data under one directory (no datetime subdirectories):* + + .. dropdown:: /var/autofs/mnt/hostmachine/projects/RRFS/prototype/met_out/metprd/point_stat + + From c00e570c8311ebbb1c9f5153f666099d168162ff Mon Sep 17 00:00:00 2001 From: bikegeek Date: Wed, 6 Nov 2024 07:45:44 -0700 Subject: [PATCH 23/57] Completed instructions/descriptions for generating the XML specification file --- docs/Users_Guide/load_data.rst | 24 ++++++++++++++++++++++-- 1 file changed, 22 insertions(+), 2 deletions(-) diff --git a/docs/Users_Guide/load_data.rst b/docs/Users_Guide/load_data.rst index 26d9629f..56bb3ea9 100644 --- a/docs/Users_Guide/load_data.rst +++ b/docs/Users_Guide/load_data.rst @@ -331,11 +331,13 @@ to the required elements and delete any optional, unused/irrelevant elements. - any digits 0-9 - _, . , - (underscore, period, dash) + **Determine whether to use folder_tmpl or load_files to define where your input data resides (descriptions below):** + *The following defines the location of the input data to be loaded into the database based on data organized by datetime (and any other criteria)* .. dropdown:: folder_tmpl - - **mandatory** only if data is organized in directories that are described by templates or if all data resides under one directory + - for data in subdirectories that are datetimes *OR* data that resides under one directory - only one folder template element is permitted (i.e. only one ... ) - **NOTE** the *date_list* element **MUST BE DEFINED** (see above in the *date_list* description) if **any subdirectories are based on datetime** @@ -546,7 +548,25 @@ to the required elements and delete any optional, unused/irrelevant elements. .. dropdown:: load_files - - **mandatory** if specifying individual data file(s) + - for specifying the location of individual data files that are in different directories + + .. dropdown:: file + + - **mandatory** for defining location of specific data files + + - maximum number of data files: 200 + + Example: + + + + /met_out/mode/mode_MASK_POLY_300000L_20120410_180000V_060000A_cts.txt + + /met_out/rhist/ensemble_stat_RRFS_GEFS_GF.SPP.SPPT_RETOP_MRMS_20220507_120000V.stat + + + + From 4ba748f25a84bbb68ed9e66b8070d71b764d069c Mon Sep 17 00:00:00 2001 From: bikegeek Date: Wed, 6 Nov 2024 11:54:29 -0700 Subject: [PATCH 24/57] all content added for Background, Create the XMO Specification File, and Troubleshooting sections --- docs/Users_Guide/load_data.rst | 134 ++++++++++++++++++++++----------- 1 file changed, 89 insertions(+), 45 deletions(-) diff --git a/docs/Users_Guide/load_data.rst b/docs/Users_Guide/load_data.rst index 56bb3ea9..337f14c5 100644 --- a/docs/Users_Guide/load_data.rst +++ b/docs/Users_Guide/load_data.rst @@ -2,7 +2,7 @@ Background ********** -The METdbLoad module provides support for inserting MET output data into a relational database +The METdbLoad module provides support for inserting MET output data (as .stat files) into a relational database (mysql, mariadb, or aurora). Before using the METdbLoad module, the database **must** exist and have the proper permissions @@ -45,9 +45,10 @@ loading the data into the database. Create your own XML specification file by copying the example specification file *METdataio/METdbLoad/test/Examples/example_load_specification.xml* file to a -location in your workspace. This file will contain the username and password to the database. +location in your workspace. -**Do not save this XML specification file where it can be read by anyone who should not have access to this information.** +**Do not save this XML specification file where it can be read by anyone who should not have access to this information, +as this file contains the database password.** .. code-block:: ini @@ -58,8 +59,17 @@ location in your workspace. This file will contain the username and password to - Replace the *path-to-your-dir* with the actual path to where this file is to be saved. -Change directory to the location where the *example_load_specification.xml* file was copied. Make the necessary edits -to the required elements and delete any optional, unused/irrelevant elements. +Change directory to the location where the *example_load_specification.xml* file was copied. + +.. code-block:: ini + + cd path-to-your-dir/load_specification.xml + +- replace *path-to-your-dir* with the full path where you are saving the XML specification file + +Using a text editor of your choice, make the necessary edits +to the required elements and delete any optional, unused/irrelevant elements, based on the explanation below (click to +expand). Remember to update the username and password that is applicable to your database. .. dropdown:: The following is an explanation of the required and optional elements and any limitations @@ -546,7 +556,7 @@ to the required elements and delete any optional, unused/irrelevant elements. *The following defines the location of specific input data files to be loaded into the database* - .. dropdown:: load_files + .. dropdown:: load_files - for specifying the location of individual data files that are in different directories @@ -566,81 +576,102 @@ to the required elements and delete any optional, unused/irrelevant elements. + *The following describe the linetypes to load* + .. dropdown:: line_type + - **optional** + - which MET output linetypes to load + - if omitted, then **all linetypes** will be loaded + - maximum number of line_type elements: 1 + *val* element defines which MET output linetypes to load -Load Data -========= + .. dropdown:: val -Now the MET data can be loaded in the database using the *met_db_load.py* script in the path-to-METdataio-source/METdbLoad/ush -directory. The *path-to-METdataio-source* is the directory where the METdataio source code is saved. + - the MET output linetype -.. code-block:: ini + - linetype name (refer to the MET User's Guide for a complete list of linetypes) - cd /path-to-METdataio-source/METdataio/METdbLoad/ush - * Replace path-to-METdataio-source to the location where the METdataio source code is saved. + Example: - python met_db_load.py /path-to/load_met.xml + - * Replace the path-to with the location where the load_met.xml file was saved. This is the same directory - you created to save the copy of the data_loading_config.yaml file. + ECNT -The usage statement: + VL1L2 -.. code-block:: ini + SAL1L2 + + + + *The following option allows one to indicate whether to save the XML commands into the database* + + .. dropdown:: load_xml + + - **optional** + - Option to save the XML into the database + - Only takes effect when the *load_note* element is present + - Acceptable values: TRUE or FALSE (case-insensitive) + - Default value: TRUE + + Example: - INFO:root:--- *** --- Start METdbLoad --- *** --- + true - usage: python met_db_load.py [-h] [-index] xmlfile [tmpdir [tmpdir ...]] - positional arguments: - xmlfile Please provide required xml load_spec filename - tmpdir Optional - when different directory wanted for tmp file + *The following allows one to create a note into the instance_info database table* - optional arguments: - -h, --help show this help message and exit - -index Only process index, do not load data + .. dropdown:: load_note + + - **optional** + - Add a descriptive "note" into the database -The **xmlfile** is the XML specification file that passes information about the MET output files to load -into the database to METdbload. It is an XML file whose top-level -tag is and it contains the following elements, divided into -functional sections: + Example: + Load HREF and RTPS data from Spring 2022 +Load Data +========= - * **:** A list structure containing individual MET output - files to load into the database. +Now the MET data can be loaded in the database using the *met_db_load.py* script in the path-to-METdataio-source/METdbLoad/ush +directory. The *path-to-METdataio-source* is the directory where the METdataio source code is saved. - * **:** Follows the list of files after the previous - tag, to end the list. +**NOTE** + **Only data files with the *.stat* extension will be loaded** - * **:** Contains a single MET output file to load. +.. dropdown:: The usage statement for met_db_load.py: + .. code-block:: ini - * **:** A template value, its name is specified by the attribute name, and its values are specified by its children **** tags. + INFO:root:--- *** --- Start METdbLoad --- *** --- - * **:** A single template value which will slot into the template in the value specified by the parent field's name. + usage: python met_db_load.py [-h] [-index] xmlfile [tmpdir [tmpdir ...]] - * **:** Specifies a previously declared **** element, using the name attribute, which represents a list of dates in a particular format. + positional arguments: + xmlfile Please provide required xml load_spec filename + tmpdir Optional - when different directory wanted for tmp file - * **:** A list structure containing the MET output file line types to load. If omitted, all line types are loaded. + optional arguments: + -h, --help show this help message and exit - * **:** Contains a single MET output file line type to be loaded, for example, CNT. + -index Only process index, do not load data - * **:** If present, creates a record in the instance_info - database database table with a note containing the body of this tag - * **: TRUE** or **FALSE**, this option indicates whether or - not to save the load xml; only effective if **** is present - - default: TRUE +.. code-block:: ini + cd /path-to-METdataio-source/METdataio/METdbLoad/ush + python met_db_load.py /path-to/load_specification.xml +- Replace *path-to-METdataio-source* to the location where the METdataio source code is saved. +- Replace the *path-to* with the location where the load_specification.xml XML specification file was saved. +Refer to the section **Create the XML Specification File** and expand the drop-down instructions +"The following is an explanation of the required and optional elements and any limitations" for details on what is expected in your XML specification file. @@ -677,3 +708,16 @@ Troubleshooting * - Solution: - This error is caused when attempting to load data into a database that does not exist. You will need to create the database, set up the appropriate privileges as outlined above, and load the schema using the mv_mysql.sql file. + + * - Error: + - /full-path-to/xyz.xml is not valid and may contain a recursive payload or an excessively large payload + + * - Solution: + - This error is typically encountered when one of the following conditions exist as a result of failing the XML validation step: + + - the order of the elements in your XML specification file is inconsistent with the order expected + - your XML specification file is missing one or more mandatory elements + - one or more elements has exceeded size limits specified in the XML schema + - you have addition XML elements that are not expected + + **Refer to the section **Create the XML Specification File** to verify that your XML specification file is correct.** From 486395cfbe115801349812b413559a8b39820e8a Mon Sep 17 00:00:00 2001 From: bikegeek Date: Wed, 6 Nov 2024 12:13:26 -0700 Subject: [PATCH 25/57] Added instructions for using mv_mysql.sql --- docs/Users_Guide/load_data.rst | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/docs/Users_Guide/load_data.rst b/docs/Users_Guide/load_data.rst index 337f14c5..c15ae0b8 100644 --- a/docs/Users_Guide/load_data.rst +++ b/docs/Users_Guide/load_data.rst @@ -10,6 +10,19 @@ Before using the METdbLoad module, the database **must** exist and have the prop line client (https://dev.mysql.com/doc/refman/en/mysql.html) to create the tables corresponding to the MET line types. The *mv_mysql.sql* file is located in the METdataio/METdbLoad/sql/ directory. +.. dropdown:: Create MET linetype tables + + mysql -udbuser -pdbpasswd dbtable_name Date: Wed, 6 Nov 2024 14:19:17 -0700 Subject: [PATCH 26/57] Update docs/Users_Guide/load_data.rst Co-authored-by: Julie Prestopnik --- docs/Users_Guide/load_data.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Users_Guide/load_data.rst b/docs/Users_Guide/load_data.rst index c15ae0b8..076974bb 100644 --- a/docs/Users_Guide/load_data.rst +++ b/docs/Users_Guide/load_data.rst @@ -731,6 +731,6 @@ Troubleshooting - the order of the elements in your XML specification file is inconsistent with the order expected - your XML specification file is missing one or more mandatory elements - one or more elements has exceeded size limits specified in the XML schema - - you have addition XML elements that are not expected + - there are additional XML elements that are not expected **Refer to the section **Create the XML Specification File** to verify that your XML specification file is correct.** From 319f1e5a74de44a0786cbb6c2152edefed6e122c Mon Sep 17 00:00:00 2001 From: bikegeek <3753118+bikegeek@users.noreply.github.com> Date: Wed, 6 Nov 2024 14:24:09 -0700 Subject: [PATCH 27/57] Update docs/Users_Guide/load_data.rst Co-authored-by: Julie Prestopnik --- docs/Users_Guide/load_data.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Users_Guide/load_data.rst b/docs/Users_Guide/load_data.rst index 076974bb..1230d8b0 100644 --- a/docs/Users_Guide/load_data.rst +++ b/docs/Users_Guide/load_data.rst @@ -37,7 +37,7 @@ XML schema and conform to size and number of element limitations. .. literalinclude:: ../../METdbLoad/test/Examples/example_load_specification.xml -.. dropdown:: This is another **example XML specification file** that is **valid**: +.. dropdown:: Another **example XML specification file** that is **valid** For specifying a list of input data files using the load_files element: From cc517ef8aebcdd073823a87a333f745b0b16aa76 Mon Sep 17 00:00:00 2001 From: bikegeek <3753118+bikegeek@users.noreply.github.com> Date: Wed, 6 Nov 2024 14:24:29 -0700 Subject: [PATCH 28/57] Update docs/Users_Guide/load_data.rst Co-authored-by: Julie Prestopnik --- docs/Users_Guide/load_data.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Users_Guide/load_data.rst b/docs/Users_Guide/load_data.rst index 1230d8b0..7fe39c54 100644 --- a/docs/Users_Guide/load_data.rst +++ b/docs/Users_Guide/load_data.rst @@ -31,7 +31,7 @@ payloads that can compromise the loading of data. The elements in the XML specification file **must** adhere to the **order specified** by the XML schema and conform to size and number of element limitations. -.. dropdown:: This is an **example XML specification file** that is **valid**: +.. dropdown:: An **example XML specification file** that is **valid** For defining data organized by dates using the folder_tmpl element: From 1c3ba24a92b1e18bdfe87dd1357d5bb3445798af Mon Sep 17 00:00:00 2001 From: bikegeek <3753118+bikegeek@users.noreply.github.com> Date: Wed, 6 Nov 2024 14:24:47 -0700 Subject: [PATCH 29/57] Update docs/Users_Guide/load_data.rst Co-authored-by: Julie Prestopnik --- docs/Users_Guide/load_data.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Users_Guide/load_data.rst b/docs/Users_Guide/load_data.rst index 7fe39c54..eeca183a 100644 --- a/docs/Users_Guide/load_data.rst +++ b/docs/Users_Guide/load_data.rst @@ -45,7 +45,7 @@ XML schema and conform to size and number of element limitations. -.. dropdown:: This is the **XML schema** that is used to **validate the XML specification file**: +.. dropdown:: The **XML schema** that is used to **validate the XML specification file** .. literalinclude:: ../../METdbLoad/ush/load_specification_schema.xsd From a6dd022e5e7f6d883e159eca32a193ee716de287 Mon Sep 17 00:00:00 2001 From: bikegeek <3753118+bikegeek@users.noreply.github.com> Date: Wed, 6 Nov 2024 14:25:01 -0700 Subject: [PATCH 30/57] Update docs/Users_Guide/load_data.rst Co-authored-by: Julie Prestopnik --- docs/Users_Guide/load_data.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Users_Guide/load_data.rst b/docs/Users_Guide/load_data.rst index eeca183a..324d7ab1 100644 --- a/docs/Users_Guide/load_data.rst +++ b/docs/Users_Guide/load_data.rst @@ -78,7 +78,7 @@ Change directory to the location where the *example_load_specification.xml* file cd path-to-your-dir/load_specification.xml -- replace *path-to-your-dir* with the full path where you are saving the XML specification file +- Replace *path-to-your-dir* with the full path where the XML specification file will be saved. Using a text editor of your choice, make the necessary edits to the required elements and delete any optional, unused/irrelevant elements, based on the explanation below (click to From b73bba9fdc75b17e48753f9c137fa6b3461bd498 Mon Sep 17 00:00:00 2001 From: bikegeek <3753118+bikegeek@users.noreply.github.com> Date: Wed, 6 Nov 2024 14:25:18 -0700 Subject: [PATCH 31/57] Update docs/Users_Guide/load_data.rst Co-authored-by: Julie Prestopnik --- docs/Users_Guide/load_data.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Users_Guide/load_data.rst b/docs/Users_Guide/load_data.rst index 324d7ab1..76318487 100644 --- a/docs/Users_Guide/load_data.rst +++ b/docs/Users_Guide/load_data.rst @@ -70,7 +70,7 @@ as this file contains the database password.** - $METDATAIO is the path to the location of the cloned or forked METdataio source code. -- Replace the *path-to-your-dir* with the actual path to where this file is to be saved. +- Replace the *path-to-your-dir* with the actual path to where this file will be saved. Change directory to the location where the *example_load_specification.xml* file was copied. From 8933bcb7c231acc8f7f36677395ab0ed52cf676e Mon Sep 17 00:00:00 2001 From: bikegeek <3753118+bikegeek@users.noreply.github.com> Date: Wed, 6 Nov 2024 14:29:05 -0700 Subject: [PATCH 32/57] Update docs/Users_Guide/load_data.rst Co-authored-by: Julie Prestopnik --- docs/Users_Guide/load_data.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/Users_Guide/load_data.rst b/docs/Users_Guide/load_data.rst index 76318487..fc53775f 100644 --- a/docs/Users_Guide/load_data.rst +++ b/docs/Users_Guide/load_data.rst @@ -30,6 +30,7 @@ that the file is valid. This validation is necessary to prevent extremely large payloads that can compromise the loading of data. The elements in the XML specification file **must** adhere to the **order specified** by the XML schema and conform to size and number of element limitations. +Click to expand the example XML specifications and the XML validation schema below. .. dropdown:: An **example XML specification file** that is **valid** From f84ffd9eb69bbf31cd238c6965f539517dd87c62 Mon Sep 17 00:00:00 2001 From: bikegeek <3753118+bikegeek@users.noreply.github.com> Date: Wed, 6 Nov 2024 14:29:20 -0700 Subject: [PATCH 33/57] Update docs/Users_Guide/load_data.rst Co-authored-by: Julie Prestopnik --- docs/Users_Guide/load_data.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Users_Guide/load_data.rst b/docs/Users_Guide/load_data.rst index fc53775f..b1ba356c 100644 --- a/docs/Users_Guide/load_data.rst +++ b/docs/Users_Guide/load_data.rst @@ -89,7 +89,7 @@ expand). Remember to update the username and password that is applicable to your *These are element names. The XML angle brackets (<>) as seen in the XML specification file are omitted for simplicity* - **!!!NOTE!!!** + ..note:: The **order of the elements** in the XML specification file is crucial. **DO NOT** modify the order of the following elements. Indentation is used to indicate hierarchical relationships between elements. From bbef41066bebbf8010afaa98e6303bfff61f3699 Mon Sep 17 00:00:00 2001 From: bikegeek <3753118+bikegeek@users.noreply.github.com> Date: Wed, 6 Nov 2024 14:29:54 -0700 Subject: [PATCH 34/57] Update docs/Users_Guide/load_data.rst Co-authored-by: Julie Prestopnik --- docs/Users_Guide/load_data.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/Users_Guide/load_data.rst b/docs/Users_Guide/load_data.rst index b1ba356c..0c22f9cc 100644 --- a/docs/Users_Guide/load_data.rst +++ b/docs/Users_Guide/load_data.rst @@ -91,7 +91,8 @@ expand). Remember to update the username and password that is applicable to your ..note:: - The **order of the elements** in the XML specification file is crucial. **DO NOT** modify the order of the following elements. + + The **order of the elements** in the XML specification file is crucial. **DO NOT** modify the order of the following elements. Indentation is used to indicate hierarchical relationships between elements. .. dropdown:: load_spec From 5f33758d7f2eef818dcc2c001442f66d1e4c1e30 Mon Sep 17 00:00:00 2001 From: bikegeek <3753118+bikegeek@users.noreply.github.com> Date: Wed, 6 Nov 2024 14:30:25 -0700 Subject: [PATCH 35/57] Update docs/Users_Guide/load_data.rst Co-authored-by: Julie Prestopnik --- docs/Users_Guide/load_data.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Users_Guide/load_data.rst b/docs/Users_Guide/load_data.rst index 0c22f9cc..7cd8d263 100644 --- a/docs/Users_Guide/load_data.rst +++ b/docs/Users_Guide/load_data.rst @@ -93,7 +93,7 @@ expand). Remember to update the username and password that is applicable to your The **order of the elements** in the XML specification file is crucial. **DO NOT** modify the order of the following elements. - Indentation is used to indicate hierarchical relationships between elements. + Indentation is used to indicate hierarchical relationships between elements. .. dropdown:: load_spec From c40d6cbe329bf15a9cf6c53a58dff460e5415c8b Mon Sep 17 00:00:00 2001 From: bikegeek <3753118+bikegeek@users.noreply.github.com> Date: Wed, 6 Nov 2024 14:30:43 -0700 Subject: [PATCH 36/57] Update docs/Users_Guide/load_data.rst Co-authored-by: Julie Prestopnik --- docs/Users_Guide/load_data.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Users_Guide/load_data.rst b/docs/Users_Guide/load_data.rst index 7cd8d263..1fa3aa40 100644 --- a/docs/Users_Guide/load_data.rst +++ b/docs/Users_Guide/load_data.rst @@ -183,7 +183,7 @@ expand). Remember to update the username and password that is applicable to your - METdataio sets default to False if this element is absent - *The following elements are used to define the format of multiple input data directories that are (optionally) organized by datetime* + *The following elements are used to define the format of multiple input data directories that are (optionally) organized by datetime.* .. dropdown:: date_list From 7ae9bbe6a9c15fe24bc0d34cbdb297057f13f78f Mon Sep 17 00:00:00 2001 From: bikegeek <3753118+bikegeek@users.noreply.github.com> Date: Wed, 6 Nov 2024 14:30:52 -0700 Subject: [PATCH 37/57] Update docs/Users_Guide/load_data.rst Co-authored-by: Julie Prestopnik --- docs/Users_Guide/load_data.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Users_Guide/load_data.rst b/docs/Users_Guide/load_data.rst index 1fa3aa40..f9745d2d 100644 --- a/docs/Users_Guide/load_data.rst +++ b/docs/Users_Guide/load_data.rst @@ -101,7 +101,7 @@ expand). Remember to update the username and password that is applicable to your - top-level tag/element - container for other elements that define connection information, flags, data input, etc. - *The following elements pertain to logging into the database* + *The following elements pertain to logging into the database.* .. dropdown:: connection From 6c2b019fb79ba18ab6ff61af563eea86480c47c9 Mon Sep 17 00:00:00 2001 From: bikegeek <3753118+bikegeek@users.noreply.github.com> Date: Wed, 6 Nov 2024 14:31:04 -0700 Subject: [PATCH 38/57] Update docs/Users_Guide/load_data.rst Co-authored-by: Julie Prestopnik --- docs/Users_Guide/load_data.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Users_Guide/load_data.rst b/docs/Users_Guide/load_data.rst index f9745d2d..4d793c0d 100644 --- a/docs/Users_Guide/load_data.rst +++ b/docs/Users_Guide/load_data.rst @@ -358,7 +358,7 @@ expand). Remember to update the username and password that is applicable to your **Determine whether to use folder_tmpl or load_files to define where your input data resides (descriptions below):** - *The following defines the location of the input data to be loaded into the database based on data organized by datetime (and any other criteria)* + *The following defines the location of the input data to be loaded into the database based on data organized by datetime (and any other criteria).* .. dropdown:: folder_tmpl From 2dd0a6697e33f3cee3388541c1023a250ba1136d Mon Sep 17 00:00:00 2001 From: bikegeek <3753118+bikegeek@users.noreply.github.com> Date: Wed, 6 Nov 2024 14:31:32 -0700 Subject: [PATCH 39/57] Update docs/Users_Guide/load_data.rst Co-authored-by: Julie Prestopnik --- docs/Users_Guide/load_data.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Users_Guide/load_data.rst b/docs/Users_Guide/load_data.rst index 4d793c0d..47c6a604 100644 --- a/docs/Users_Guide/load_data.rst +++ b/docs/Users_Guide/load_data.rst @@ -329,7 +329,7 @@ expand). Remember to update the username and password that is applicable to your - indicate whether or not to force load paths/files that already exist - True or False (case insensitive) - *The following elements indicate which group the database should be assigned and a description* + *The following elements indicate which group the database should be assigned and a description.* .. dropdown:: group From 8e451f739c578ed766ab01958cc14c7c97b9a304 Mon Sep 17 00:00:00 2001 From: bikegeek <3753118+bikegeek@users.noreply.github.com> Date: Wed, 6 Nov 2024 14:31:40 -0700 Subject: [PATCH 40/57] Update docs/Users_Guide/load_data.rst Co-authored-by: Julie Prestopnik --- docs/Users_Guide/load_data.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Users_Guide/load_data.rst b/docs/Users_Guide/load_data.rst index 47c6a604..6931f57e 100644 --- a/docs/Users_Guide/load_data.rst +++ b/docs/Users_Guide/load_data.rst @@ -569,7 +569,7 @@ expand). Remember to update the username and password that is applicable to your - *The following defines the location of specific input data files to be loaded into the database* + *The following defines the location of specific input data files to be loaded into the database.* .. dropdown:: load_files From 3381ee01f9aad83d67ce2fed38da558fa20ce943 Mon Sep 17 00:00:00 2001 From: bikegeek <3753118+bikegeek@users.noreply.github.com> Date: Wed, 6 Nov 2024 14:31:51 -0700 Subject: [PATCH 41/57] Update docs/Users_Guide/load_data.rst Co-authored-by: Julie Prestopnik --- docs/Users_Guide/load_data.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Users_Guide/load_data.rst b/docs/Users_Guide/load_data.rst index 6931f57e..3866d3f8 100644 --- a/docs/Users_Guide/load_data.rst +++ b/docs/Users_Guide/load_data.rst @@ -591,7 +591,7 @@ expand). Remember to update the username and password that is applicable to your - *The following describe the linetypes to load* + *The following describe the linetypes to load.* .. dropdown:: line_type From e9057abd2c56187c1af48544a45f7ad626025e29 Mon Sep 17 00:00:00 2001 From: bikegeek <3753118+bikegeek@users.noreply.github.com> Date: Wed, 6 Nov 2024 14:32:01 -0700 Subject: [PATCH 42/57] Update docs/Users_Guide/load_data.rst Co-authored-by: Julie Prestopnik --- docs/Users_Guide/load_data.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Users_Guide/load_data.rst b/docs/Users_Guide/load_data.rst index 3866d3f8..d580e7cc 100644 --- a/docs/Users_Guide/load_data.rst +++ b/docs/Users_Guide/load_data.rst @@ -621,7 +621,7 @@ expand). Remember to update the username and password that is applicable to your - *The following option allows one to indicate whether to save the XML commands into the database* + *The following option allows one to indicate whether to save the XML commands into the database.* .. dropdown:: load_xml From d0dbb3a216073b506e2afc0a23ab5d5f6c6d549a Mon Sep 17 00:00:00 2001 From: bikegeek <3753118+bikegeek@users.noreply.github.com> Date: Wed, 6 Nov 2024 14:32:10 -0700 Subject: [PATCH 43/57] Update docs/Users_Guide/load_data.rst Co-authored-by: Julie Prestopnik --- docs/Users_Guide/load_data.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Users_Guide/load_data.rst b/docs/Users_Guide/load_data.rst index d580e7cc..e1c74b1e 100644 --- a/docs/Users_Guide/load_data.rst +++ b/docs/Users_Guide/load_data.rst @@ -636,7 +636,7 @@ expand). Remember to update the username and password that is applicable to your true - *The following allows one to create a note into the instance_info database table* + *The following allows one to create a note into the instance_info database table.* .. dropdown:: load_note From 187247bfbfe4e6ae42bfef7fce008a054ce2b3e9 Mon Sep 17 00:00:00 2001 From: bikegeek <3753118+bikegeek@users.noreply.github.com> Date: Wed, 6 Nov 2024 14:32:23 -0700 Subject: [PATCH 44/57] Update docs/Users_Guide/load_data.rst Co-authored-by: Julie Prestopnik --- docs/Users_Guide/load_data.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Users_Guide/load_data.rst b/docs/Users_Guide/load_data.rst index e1c74b1e..db475efc 100644 --- a/docs/Users_Guide/load_data.rst +++ b/docs/Users_Guide/load_data.rst @@ -730,7 +730,7 @@ Troubleshooting * - Solution: - This error is typically encountered when one of the following conditions exist as a result of failing the XML validation step: - - the order of the elements in your XML specification file is inconsistent with the order expected + - the order of the elements in the XML specification file is inconsistent with the order expected - your XML specification file is missing one or more mandatory elements - one or more elements has exceeded size limits specified in the XML schema - there are additional XML elements that are not expected From 367fef5b22425a7a4ddaf83c8e43efacce63d670 Mon Sep 17 00:00:00 2001 From: bikegeek <3753118+bikegeek@users.noreply.github.com> Date: Wed, 6 Nov 2024 14:32:34 -0700 Subject: [PATCH 45/57] Update docs/Users_Guide/load_data.rst Co-authored-by: Julie Prestopnik --- docs/Users_Guide/load_data.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Users_Guide/load_data.rst b/docs/Users_Guide/load_data.rst index db475efc..5548c69c 100644 --- a/docs/Users_Guide/load_data.rst +++ b/docs/Users_Guide/load_data.rst @@ -731,7 +731,7 @@ Troubleshooting - This error is typically encountered when one of the following conditions exist as a result of failing the XML validation step: - the order of the elements in the XML specification file is inconsistent with the order expected - - your XML specification file is missing one or more mandatory elements + - the XML specification file is missing one or more mandatory elements - one or more elements has exceeded size limits specified in the XML schema - there are additional XML elements that are not expected From ff92e36be368e140b17cbada0aaf5922665ed5f7 Mon Sep 17 00:00:00 2001 From: bikegeek <3753118+bikegeek@users.noreply.github.com> Date: Wed, 6 Nov 2024 14:32:51 -0700 Subject: [PATCH 46/57] Update docs/Users_Guide/load_data.rst Co-authored-by: Julie Prestopnik --- docs/Users_Guide/load_data.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Users_Guide/load_data.rst b/docs/Users_Guide/load_data.rst index 5548c69c..ef501ee5 100644 --- a/docs/Users_Guide/load_data.rst +++ b/docs/Users_Guide/load_data.rst @@ -735,4 +735,4 @@ Troubleshooting - one or more elements has exceeded size limits specified in the XML schema - there are additional XML elements that are not expected - **Refer to the section **Create the XML Specification File** to verify that your XML specification file is correct.** + **Refer to the section **Create the XML Specification File** to verify that the XML specification file is correct.** From 5e24f29c749370cb7ac5fdf59d7a5e69c889fb81 Mon Sep 17 00:00:00 2001 From: bikegeek Date: Wed, 6 Nov 2024 16:12:07 -0700 Subject: [PATCH 47/57] Fixed some formatting/reorganizing and typos --- docs/Users_Guide/load_data.rst | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/docs/Users_Guide/load_data.rst b/docs/Users_Guide/load_data.rst index ef501ee5..9f3689bd 100644 --- a/docs/Users_Guide/load_data.rst +++ b/docs/Users_Guide/load_data.rst @@ -85,15 +85,15 @@ Using a text editor of your choice, make the necessary edits to the required elements and delete any optional, unused/irrelevant elements, based on the explanation below (click to expand). Remember to update the username and password that is applicable to your database. -.. dropdown:: The following is an explanation of the required and optional elements and any limitations - *These are element names. The XML angle brackets (<>) as seen in the XML specification file are omitted for simplicity* +.. note:: + The **order of the elements** in the XML specification file is crucial. **DO NOT** modify the order of the described elements (below). + + - ..note:: +.. dropdown:: The following is an explanation of the required and optional elements and any limitations - - The **order of the elements** in the XML specification file is crucial. **DO NOT** modify the order of the following elements. - Indentation is used to indicate hierarchical relationships between elements. + *These are element names. The XML angle brackets (<>) as seen in the XML specification file are omitted for simplicity. Indentation is used to indicate hierarchical relationships between elements.* .. dropdown:: load_spec @@ -358,7 +358,7 @@ expand). Remember to update the username and password that is applicable to your **Determine whether to use folder_tmpl or load_files to define where your input data resides (descriptions below):** - *The following defines the location of the input data to be loaded into the database based on data organized by datetime (and any other criteria).* + *The following defines the location of the input data to be loaded into the database based on data organized by datetime (and/or any other criteria).* .. dropdown:: folder_tmpl @@ -402,13 +402,13 @@ expand). Remember to update the username and password that is applicable to your - each *field* element defines the attribute value template in the directory structure (i.e. the variable inside the {}) - *val* elements that can specify more subdirectories - - optional + - **optional** - necessary when specifying subdirectories that are not datetimes (e.g. /path-to/model_A/air_quality/) - maximum number of val elements: 100 - *date_list* elements for subdirectories that are datetimes - - optional + - **optional** - mandatory when subdirectories are datetimes (e.g. /path-to/20240101/model/) - maximum number of date_list elements: 5 @@ -515,7 +515,7 @@ expand). Remember to update the username and password that is applicable to your - .. dropdown:: valid_times + .. dropdown:: date_list - the *name* attribute corresponds to one of the *date_list* attribute names From aebdcb6cb698db7f6014e021fff6ee0de68da214 Mon Sep 17 00:00:00 2001 From: bikegeek Date: Wed, 6 Nov 2024 16:15:57 -0700 Subject: [PATCH 48/57] Replacing NOTE with RST syntax --- docs/Users_Guide/load_data.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/Users_Guide/load_data.rst b/docs/Users_Guide/load_data.rst index 9f3689bd..4cb2feee 100644 --- a/docs/Users_Guide/load_data.rst +++ b/docs/Users_Guide/load_data.rst @@ -654,8 +654,8 @@ Load Data Now the MET data can be loaded in the database using the *met_db_load.py* script in the path-to-METdataio-source/METdbLoad/ush directory. The *path-to-METdataio-source* is the directory where the METdataio source code is saved. -**NOTE** - **Only data files with the *.stat* extension will be loaded** +.. note:: + Only data files with the *.stat* extension will be loaded .. dropdown:: The usage statement for met_db_load.py: From cadd58ca87c34b8974a63266ce0fec9e14a7b7b4 Mon Sep 17 00:00:00 2001 From: bikegeek Date: Thu, 7 Nov 2024 12:34:47 -0700 Subject: [PATCH 49/57] Clean up, move some instructions to the Using METdbLoad secton --- docs/Users_Guide/installation.rst | 27 ++++----------------------- 1 file changed, 4 insertions(+), 23 deletions(-) diff --git a/docs/Users_Guide/installation.rst b/docs/Users_Guide/installation.rst index 657c8d05..a17c1344 100644 --- a/docs/Users_Guide/installation.rst +++ b/docs/Users_Guide/installation.rst @@ -45,32 +45,13 @@ Installation Download -------- -Download METdataio into an installation directory. Programs can be run from -the ush subdirectory. +Download the latest release: .. code-block:: ini - git clone https://github.com/dtcenter/METdataio [install]/METdataio - cd [install]/METdataio/METdbLoad/ush + git clone https://github.com/dtcenter/METdataio -Create Database ---------------- +.. note:: -Data must be loaded into a database which has the prefix \'\mv_\'\, -e.g. mv_met_data. This database must be structured with the METviewer -mv_mysql.sql schema: + Programs can be run from the METdbLoad/ush subdirectory. -.. code-block:: ini - - cd [install]/metviewer - mysql -u[db_username] -p[db_password] -e'create database [db_name];' - mysql -u[db_username] -p[db_password] [db_name] < sql/mv_mysql.sql - - -Create an XML load specification document which contains information about your -MET data. Run METdbload using the XML load specification as input (called -[load_xml] here) and monitor progress: - -.. code-block:: ini - - python met_db_load.py [load_xml] [optional redirection of output, e.g. &> log/load_[date].log &]' From 853457c9cadf663756a1cca14421153185ef9ae7 Mon Sep 17 00:00:00 2001 From: bikegeek Date: Thu, 7 Nov 2024 12:35:42 -0700 Subject: [PATCH 50/57] Add a Create Database and Create an XML Load specification Document. Clean up some formatting, migrated some content from the Insatllation section to this section. --- docs/Users_Guide/load_data.rst | 93 ++++++++++++++++++++++------------ 1 file changed, 60 insertions(+), 33 deletions(-) diff --git a/docs/Users_Guide/load_data.rst b/docs/Users_Guide/load_data.rst index 4cb2feee..6d91606b 100644 --- a/docs/Users_Guide/load_data.rst +++ b/docs/Users_Guide/load_data.rst @@ -1,36 +1,59 @@ -********** -Background -********** +*************** +Using METdbLoad +*************** The METdbLoad module provides support for inserting MET output data (as .stat files) into a relational database (mysql, mariadb, or aurora). + +Create Database +--------------- + Before using the METdbLoad module, the database **must** exist and have the proper permissions -(i.e. grant privileges to insert, delete, update, and index). Next, use *mv_mysql.sql* and the mysql command -line client (https://dev.mysql.com/doc/refman/en/mysql.html) to create the tables corresponding to the MET line types. -The *mv_mysql.sql* file is located in the METdataio/METdbLoad/sql/ directory. +(i.e. grant privileges to insert, delete, update, and index). -.. dropdown:: Create MET linetype tables - mysql -udbuser -pdbpasswd dbtable_name) as seen in the XML specification file are omitted for simplicity. Indentation is used to indicate hierarchical relationships between elements.* + *These are element names. The XML angle brackets (<>) as seen in the XML load specification file are omitted for simplicity. Indentation is used to indicate hierarchical relationships between elements.* .. dropdown:: load_spec @@ -356,7 +377,7 @@ expand). Remember to update the username and password that is applicable to your - any digits 0-9 - _, . , - (underscore, period, dash) - **Determine whether to use folder_tmpl or load_files to define where your input data resides (descriptions below):** + **Determine whether to use folder_tmpl or load_files to define where the input data resides (descriptions below):** *The following defines the location of the input data to be loaded into the database based on data organized by datetime (and/or any other criteria).* @@ -455,7 +476,7 @@ expand). Remember to update the username and password that is applicable to your - .. dropdown:: The "folder_dates" name attribute was assigned in the date_list portion of the XML specification file + .. dropdown:: The "folder_dates" name attribute was assigned in the date_list portion of the XML load specification file **** @@ -525,7 +546,7 @@ expand). Remember to update the username and password that is applicable to your - .. dropdown:: the "valid_dates" name attribute value was assigned in the date_list portion of the XML specification file + .. dropdown:: the "valid_dates" name attribute value was assigned in the date_list portion of the XML load specification file **** @@ -682,11 +703,17 @@ directory. The *path-to-METdataio-source* is the directory where the METdataio python met_db_load.py /path-to/load_specification.xml +If logging is desired, redirect output to a log file (via &> command): + +.. code-block:: ini + + python met_db_load.py /path-to/load_specification.xml &> log/your_logname.log & + - Replace *path-to-METdataio-source* to the location where the METdataio source code is saved. -- Replace the *path-to* with the location where the load_specification.xml XML specification file was saved. +- Replace the *path-to* with the location where the load_specification.xml XML load specification file was saved. Refer to the section **Create the XML Specification File** and expand the drop-down instructions -"The following is an explanation of the required and optional elements and any limitations" for details on what is expected in your XML specification file. +"The following is an explanation of the required and optional elements and any limitations" for details on what is expected in the XML load specification file. @@ -730,9 +757,9 @@ Troubleshooting * - Solution: - This error is typically encountered when one of the following conditions exist as a result of failing the XML validation step: - - the order of the elements in the XML specification file is inconsistent with the order expected - - the XML specification file is missing one or more mandatory elements + - the order of the elements in the XML load specification file is inconsistent with the order expected + - the XML load specification file is missing one or more mandatory elements - one or more elements has exceeded size limits specified in the XML schema - there are additional XML elements that are not expected - **Refer to the section **Create the XML Specification File** to verify that the XML specification file is correct.** + **Refer to the section **Create the XML Load Specification File** to verify that the XML load specification file is correct.** From 5792bc4fe24d3e611a29961c12995e86c47b51f1 Mon Sep 17 00:00:00 2001 From: bikegeek Date: Thu, 7 Nov 2024 12:42:15 -0700 Subject: [PATCH 51/57] Fixed formatting, comments that were in the code block are now outside the code block --- docs/Users_Guide/load_data.rst | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/Users_Guide/load_data.rst b/docs/Users_Guide/load_data.rst index 6d91606b..d4251435 100644 --- a/docs/Users_Guide/load_data.rst +++ b/docs/Users_Guide/load_data.rst @@ -23,11 +23,12 @@ mv_mysql.sql schema: mysql -u[db_username] -p[db_password] -e'create database [db_name];' mysql -u[db_username] -p[db_password] [db_name] < sql/mv_mysql.sql - - Replace *db_usenamer* with the username - - Replace *db_passwd* with the password +- Replace *db_usenamer* with the username - - Replace *db_name* with the appropriate database name that begins with **mv** (e.g. mv_xyz ) +- Replace *db_passwd* with the password + +- Replace *db_name* with the appropriate database name that begins with **mv** (e.g. mv_xyz ) .. note:: From af475e9e06f66fe214181194ff39c1b298b4096d Mon Sep 17 00:00:00 2001 From: bikegeek Date: Thu, 7 Nov 2024 16:06:19 -0700 Subject: [PATCH 52/57] lost the last bullet point for the create database section on creating the database --- docs/Users_Guide/load_data.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/Users_Guide/load_data.rst b/docs/Users_Guide/load_data.rst index d4251435..1dee0cd0 100644 --- a/docs/Users_Guide/load_data.rst +++ b/docs/Users_Guide/load_data.rst @@ -30,6 +30,8 @@ mv_mysql.sql schema: - Replace *db_name* with the appropriate database name that begins with **mv** (e.g. mv_xyz ) +- Repace *database* with the database name + .. note:: Use the mysql command line client (https://dev.mysql.com/doc/refman/en/mysql.html) to run the above instructions. From 9a5c1493c8885683f665100d482bcc04d05533c6 Mon Sep 17 00:00:00 2001 From: bikegeek Date: Thu, 7 Nov 2024 16:12:04 -0700 Subject: [PATCH 53/57] Remove last change, the text was correct --- docs/Users_Guide/load_data.rst | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/Users_Guide/load_data.rst b/docs/Users_Guide/load_data.rst index 1dee0cd0..c9686fe2 100644 --- a/docs/Users_Guide/load_data.rst +++ b/docs/Users_Guide/load_data.rst @@ -30,7 +30,6 @@ mv_mysql.sql schema: - Replace *db_name* with the appropriate database name that begins with **mv** (e.g. mv_xyz ) -- Repace *database* with the database name .. note:: From e68d8515ac23a821062b73fa3ef5273abbd69819 Mon Sep 17 00:00:00 2001 From: bikegeek <3753118+bikegeek@users.noreply.github.com> Date: Mon, 11 Nov 2024 10:42:52 -0700 Subject: [PATCH 54/57] Update docs/Users_Guide/load_data.rst Co-authored-by: Julie Prestopnik --- docs/Users_Guide/load_data.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Users_Guide/load_data.rst b/docs/Users_Guide/load_data.rst index c9686fe2..6f22c82f 100644 --- a/docs/Users_Guide/load_data.rst +++ b/docs/Users_Guide/load_data.rst @@ -93,7 +93,7 @@ as this file contains the database password.** cp $METDATAIO_HOME/METdbLoad/test/Examples/example_load_specification.xml path-to-your-dir/load_specification.xml -- $METDATAIO is the path to the location of the cloned or forked METdataio source code. +- $METDATAIO_HOME is the path to the location of the cloned or forked METdataio source code. - Replace the *path-to-your-dir* with the actual path to where this file will be saved. From 81458e93fc00d66d582e198fda9ccb3c87d87f54 Mon Sep 17 00:00:00 2001 From: bikegeek <3753118+bikegeek@users.noreply.github.com> Date: Mon, 11 Nov 2024 13:10:30 -0700 Subject: [PATCH 55/57] Update docs/Users_Guide/load_data.rst Co-authored-by: Julie Prestopnik --- docs/Users_Guide/load_data.rst | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/Users_Guide/load_data.rst b/docs/Users_Guide/load_data.rst index 6f22c82f..4d4f8ce7 100644 --- a/docs/Users_Guide/load_data.rst +++ b/docs/Users_Guide/load_data.rst @@ -43,6 +43,9 @@ mv_mysql.sql schema: + - Replace *dbuser* with the username + - Replace *dbpasswd* with the password + - Replace *dbtable_name* with the appropriate database name that begins with **mv** (e.g. mv_xyz ) - Replace *path-to-METdataio* with the path where the METdataio source code is saved - Note the use of the redirection symbol '<' in the command From 86ecdef0ff616b83002fec3b50350c0b2f8a99e5 Mon Sep 17 00:00:00 2001 From: bikegeek Date: Mon, 11 Nov 2024 13:17:54 -0700 Subject: [PATCH 56/57] manually modify instructions to indicate path to location of cloned data --- docs/Users_Guide/load_data.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Users_Guide/load_data.rst b/docs/Users_Guide/load_data.rst index 4d4f8ce7..5151cd85 100644 --- a/docs/Users_Guide/load_data.rst +++ b/docs/Users_Guide/load_data.rst @@ -98,7 +98,7 @@ as this file contains the database password.** - $METDATAIO_HOME is the path to the location of the cloned or forked METdataio source code. -- Replace the *path-to-your-dir* with the actual path to where this file will be saved. +- Replace the *path-to-your-dir* with the path to the location of the cloned or forked METdataio source code. Change directory to the location where the *example_load_specification.xml* file was copied. From 627003bf8e6469523bcd4de734a6634a3a56d5c3 Mon Sep 17 00:00:00 2001 From: bikegeek Date: Mon, 11 Nov 2024 13:21:20 -0700 Subject: [PATCH 57/57] Fixed instructions to use same language as earlier content for consistency --- docs/Users_Guide/load_data.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/Users_Guide/load_data.rst b/docs/Users_Guide/load_data.rst index 5151cd85..2625d05f 100644 --- a/docs/Users_Guide/load_data.rst +++ b/docs/Users_Guide/load_data.rst @@ -98,7 +98,7 @@ as this file contains the database password.** - $METDATAIO_HOME is the path to the location of the cloned or forked METdataio source code. -- Replace the *path-to-your-dir* with the path to the location of the cloned or forked METdataio source code. +- Replace the *path-to-your-dir* with the actual path to where this file will be saved. Change directory to the location where the *example_load_specification.xml* file was copied. @@ -714,7 +714,7 @@ If logging is desired, redirect output to a log file (via &> command): python met_db_load.py /path-to/load_specification.xml &> log/your_logname.log & -- Replace *path-to-METdataio-source* to the location where the METdataio source code is saved. +- Replace *path-to-METdataio-source* to the path to the location of the cloned or forked METdataio source code. - Replace the *path-to* with the location where the load_specification.xml XML load specification file was saved. Refer to the section **Create the XML Specification File** and expand the drop-down instructions