From a5b65abba488080e43658e9c989ff5533d38edc5 Mon Sep 17 00:00:00 2001
From: bikegeek <3753118+bikegeek@users.noreply.github.com>
Date: Wed, 23 Oct 2024 14:56:28 -0600
Subject: [PATCH] Feature internal 56 me tdataio validate payload (#345)
* Internal issue #56 XML schema for validating to prevent DoS via large payload, recursive payloads, ill-formed XML specification file
* schema for validating the payload (recursive, excessively large)
* Delete METdbLoad/ush/specification_schema.xsd
* internal issue #56 validating payload using XML schema
* internal issue #56 invalid XML spec files used to test XML validation
* internal issue #56 add the location of the XML schema file to be used in validating the XML specification file
* internal issue #56 tests added to verify validation code is providing expected results
* internal issue#56 added new fixture used in testing XML validation code
* fix import for read_load_xml module
* removed extraneous ',' in import
* Working version but still needs to check for recursive payloads for some elements
* Valid XML that is used for real-world data and is valid XML
* Change the name of the XML schema file
* Use the full_example.xml file instead of the test_load_specification.xml for testing against a valid XML file
* Delete METdbLoad/ush/load_specification.xsd
* Test for recursive payload in load_val fields
* Added test for recursion under the load_val complex type
* Change values to prevent recursive payloads and remove defunct regex
* Remove unused imports, add test for recursion under the load_val fields
* Add some extra elements
* Add more recursive elements to trigger ValueError
* skip testing the recursion in load_val
* Remove limit to number of load_val elements
* Fixed incorrect skip syntax
* Reinstate the maxOccurs and minOccurs for the field
* Update temporary XML spec file to match load_specification_schema.xsd
* Config file for testing recursive payload in the fields element
* Work-in-progress. Recursive payloads checked for some elements but no checking for large payloads
* Added test for recursive payload for fields (in addition to test for recursive val elements)
* Update tests and test config files
* Additional test configuration files
* updated schema, now working
* modified test configuration file
* Allow '-' in regex for limited string type
* Updated file so it is valid with respect to the schema
* Include testing one of the xml specification files used in testing two databases
* Reformat code for easier reading, update the load_met_gha_new.xml file to be valid
* include testing the load_met_gha_new.xml file
* Updated: reformatted and updated to conform to schema
* Added an extra date_list element
* added testing xml specification file with more than one date_list
* Clean up unnecessary comments
* Update number of date_list items
* Explicitly set minLength and maxLength for hostname, db name, password, etc.
* allow password to be string type and limit length of password
* comment out mysql commands. ci-run-all-cases
* Fix comment
* Remove main function with hard-coded paths. Only useful during development.
* Add support for line_type element
* Added line_type element and some values for example xml specification
---
METdbLoad/test/full_example.xml | 6 ++++++
METdbLoad/ush/load_specification_schema.xsd | 17 +++++++++++++++--
2 files changed, 21 insertions(+), 2 deletions(-)
diff --git a/METdbLoad/test/full_example.xml b/METdbLoad/test/full_example.xml
index 6151ee9..b393bfe 100644
--- a/METdbLoad/test/full_example.xml
+++ b/METdbLoad/test/full_example.xml
@@ -59,6 +59,12 @@
grid_stat_cmn
point_stat_cmn
+
+ CNT
+ ORANK
+ SEEPS_MPR
+ ME
+
true
diff --git a/METdbLoad/ush/load_specification_schema.xsd b/METdbLoad/ush/load_specification_schema.xsd
index 1128035..4e93fd3 100644
--- a/METdbLoad/ush/load_specification_schema.xsd
+++ b/METdbLoad/ush/load_specification_schema.xsd
@@ -81,6 +81,14 @@
+
+
+
+
+
+
+
+
@@ -136,7 +144,6 @@
-
@@ -147,11 +154,17 @@
-
+
+
+
+
+
+
+