Feature internal 56 me tdataio validate payload (#345)

* Internal issue #56 XML schema for validating to prevent DoS via large payload, recursive payloads, ill-formed XML specification file

* schema for validating the payload (recursive, excessively large)

* Delete METdbLoad/ush/specification_schema.xsd

* internal issue #56 validating payload using XML schema

* internal issue #56 invalid XML spec files used to test XML validation

* internal issue #56 add the location of the XML schema file to be used in validating the XML specification file

* internal issue #56 tests added to verify validation code is providing expected results

* internal issue#56 added new fixture used in testing XML validation code

* fix import for read_load_xml module

* removed extraneous ',' in import

* Working version but still needs to check for recursive payloads for some elements

* Valid XML that is used for real-world data and is valid XML

* Change the name of the XML schema file

* Use the full_example.xml file instead of the test_load_specification.xml for testing against a valid XML file

* Delete METdbLoad/ush/load_specification.xsd

* Test for recursive payload in load_val fields

* Added test for recursion under the load_val complex type

* Change values to prevent recursive payloads and remove defunct regex

* Remove unused imports, add test for recursion under the load_val fields

* Add some extra elements

* Add more recursive elements to trigger ValueError

* skip testing the recursion in load_val

* Remove limit to number of load_val elements

* Fixed incorrect skip syntax

* Reinstate the maxOccurs and minOccurs for the field

* Update temporary XML spec file to match load_specification_schema.xsd

* Config file for testing recursive payload in the fields element

* Work-in-progress.  Recursive payloads checked for some elements but no checking for large payloads

* Added test for recursive payload for fields (in addition to test for recursive val elements)

* Update tests and test config files

* Additional test configuration files

* updated schema, now working

* modified test configuration file

* Allow '-' in regex for limited string type

* Updated file so it is valid with respect to the schema

* Include testing one of the xml specification files used in testing two databases

* Reformat code for easier reading, update the load_met_gha_new.xml file to be valid

* include testing the load_met_gha_new.xml file

* Updated: reformatted and updated to conform to schema

* Added an extra date_list element

* added testing xml specification file with more than one date_list

* Clean up unnecessary comments

* Update number of date_list items

* Explicitly set minLength and maxLength for hostname, db name, password, etc.

* allow password to be string type and limit length of password

* comment out mysql commands. ci-run-all-cases

* Fix comment

* Remove main function with hard-coded paths. Only useful during development.

* Add support for line_type element

* Added line_type element and some values for example xml specification

METdbLoad/test/full_example.xml

@@ -59,6 +59,12 @@
       <val>grid_stat_cmn</val>
       <val>point_stat_cmn</val>
     </field>
+    <line_type>
+      <val>CNT</val>
+      <val>ORANK</val>
+      <val>SEEPS_MPR</val>
+      <val>ME</val>
+    </line_type>
   </load_val>
 
   <load_xml>true</load_xml>

METdbLoad/ush/load_specification_schema.xsd

@@ -81,6 +81,14 @@
         </xs:restriction>
     </xs:simpleType>
 
+    <xs:simpleType name="linetype">
+        <xs:restriction base="xs:string">
+            <xs:minLength value="2"/>
+            <xs:maxLength value="25"/>
+            <xs:pattern value="[_a-zA-Z0-9]+"/>
+        </xs:restriction>
+    </xs:simpleType>
+
 
     <!-- COMPLEX TYPES -->
     <xs:element name="load_spec">
@@ -136,7 +144,6 @@
                             <xs:element name="field" minOccurs="0" maxOccurs="20">
                                 <xs:complexType>
                                     <xs:sequence>
-
                                         <xs:element name="date_list" minOccurs="0" maxOccurs="5">
                                             <xs:complexType>
                                                 <xs:simpleContent>
@@ -147,11 +154,17 @@
                                             </xs:complexType>
                                         </xs:element>
                                          <xs:element type="xs:string" name="val" minOccurs="0" maxOccurs="100"/>
-
                                     </xs:sequence>
                                     <xs:attribute type="xs:string" name="name"/>
                                 </xs:complexType>
                             </xs:element>
+                            <xs:element name="line_type" minOccurs="0" maxOccurs="1">
+                               <xs:complexType>
+                                  <xs:sequence>
+                                     <xs:element type="xs:string" name="val" maxOccurs="800" minOccurs="0"/>
+                                  </xs:sequence>
+                               </xs:complexType>
+                            </xs:element>
                         </xs:sequence>
                     </xs:complexType>
                 </xs:element>