From a5b65abba488080e43658e9c989ff5533d38edc5 Mon Sep 17 00:00:00 2001 From: bikegeek <3753118+bikegeek@users.noreply.github.com> Date: Wed, 23 Oct 2024 14:56:28 -0600 Subject: [PATCH] Feature internal 56 me tdataio validate payload (#345) * Internal issue #56 XML schema for validating to prevent DoS via large payload, recursive payloads, ill-formed XML specification file * schema for validating the payload (recursive, excessively large) * Delete METdbLoad/ush/specification_schema.xsd * internal issue #56 validating payload using XML schema * internal issue #56 invalid XML spec files used to test XML validation * internal issue #56 add the location of the XML schema file to be used in validating the XML specification file * internal issue #56 tests added to verify validation code is providing expected results * internal issue#56 added new fixture used in testing XML validation code * fix import for read_load_xml module * removed extraneous ',' in import * Working version but still needs to check for recursive payloads for some elements * Valid XML that is used for real-world data and is valid XML * Change the name of the XML schema file * Use the full_example.xml file instead of the test_load_specification.xml for testing against a valid XML file * Delete METdbLoad/ush/load_specification.xsd * Test for recursive payload in load_val fields * Added test for recursion under the load_val complex type * Change values to prevent recursive payloads and remove defunct regex * Remove unused imports, add test for recursion under the load_val fields * Add some extra elements * Add more recursive elements to trigger ValueError * skip testing the recursion in load_val * Remove limit to number of load_val elements * Fixed incorrect skip syntax * Reinstate the maxOccurs and minOccurs for the field * Update temporary XML spec file to match load_specification_schema.xsd * Config file for testing recursive payload in the fields element * Work-in-progress. Recursive payloads checked for some elements but no checking for large payloads * Added test for recursive payload for fields (in addition to test for recursive val elements) * Update tests and test config files * Additional test configuration files * updated schema, now working * modified test configuration file * Allow '-' in regex for limited string type * Updated file so it is valid with respect to the schema * Include testing one of the xml specification files used in testing two databases * Reformat code for easier reading, update the load_met_gha_new.xml file to be valid * include testing the load_met_gha_new.xml file * Updated: reformatted and updated to conform to schema * Added an extra date_list element * added testing xml specification file with more than one date_list * Clean up unnecessary comments * Update number of date_list items * Explicitly set minLength and maxLength for hostname, db name, password, etc. * allow password to be string type and limit length of password * comment out mysql commands. ci-run-all-cases * Fix comment * Remove main function with hard-coded paths. Only useful during development. * Add support for line_type element * Added line_type element and some values for example xml specification --- METdbLoad/test/full_example.xml | 6 ++++++ METdbLoad/ush/load_specification_schema.xsd | 17 +++++++++++++++-- 2 files changed, 21 insertions(+), 2 deletions(-) diff --git a/METdbLoad/test/full_example.xml b/METdbLoad/test/full_example.xml index 6151ee9..b393bfe 100644 --- a/METdbLoad/test/full_example.xml +++ b/METdbLoad/test/full_example.xml @@ -59,6 +59,12 @@ grid_stat_cmn point_stat_cmn + + CNT + ORANK + SEEPS_MPR + ME + true diff --git a/METdbLoad/ush/load_specification_schema.xsd b/METdbLoad/ush/load_specification_schema.xsd index 1128035..4e93fd3 100644 --- a/METdbLoad/ush/load_specification_schema.xsd +++ b/METdbLoad/ush/load_specification_schema.xsd @@ -81,6 +81,14 @@ + + + + + + + + @@ -136,7 +144,6 @@ - @@ -147,11 +154,17 @@ - + + + + + + +