Business logic validator plugin implementation

README.md

@@ -121,7 +121,7 @@ To run the schema validations, the command `schema-enforcer validate` can be run
 
 ```shell
 bash$ schema-enforcer validate
-schema-enforcer validate            
+schema-enforcer validate
 ALL SCHEMA VALIDATION CHECKS PASSED
 ```
 
@@ -140,14 +140,14 @@ If we modify one of the addresses in the `chi-beijing-rt1/dns.yml` file so that
 
 ```yaml
 bash$ cat chi-beijing-rt1/dns.yml
-# jsonschema: schemas/dns_servers       
+# jsonschema: schemas/dns_servers
 ---
 dns_servers:
   - address: true
   - address: "10.2.2.2"
 ```
 ```shell
-bash$ test-schema validate            
+bash$ test-schema validate
 FAIL | [ERROR] True is not of type 'string' [FILE] ./chi-beijing-rt1/dns.yml [PROPERTY] dns_servers:0:address
 bash$ echo $?
 1
@@ -160,7 +160,7 @@ When a structured data file fails schema validation, `schema-enforcer` exits wit
 Schema enforcer will work with default settings, however, a `pyproject.toml` file can be placed at the root of the path in which `schema-enforcer` is run in order to override default settings or declare configuration for more advanced features. Inside of this `pyproject.toml` file, `tool.schema_enfocer` sections can be used to declare settings for schema enforcer. Take for example the `pyproject.toml` file in example 2.
 
 ```shell
-bash$ cd examples/example2 && tree -L 2         
+bash$ cd examples/example2 && tree -L 2
 .
 ├── README.md
 ├── hostvars
@@ -198,3 +198,4 @@ Detailed documentation can be found in the README.md files inside of the `docs/`
 - [The `validate` command](docs/validate_command.md)
 - [Mapping Structured Data Files to Schema Files](docs/mapping_schemas.md)
 - [The `schema` command](docs/schema_command.md)
+- [Implementing custom validators](docs/custom_validators.md)

docs/custom_validators.md

@@ -0,0 +1,158 @@
+# Implementing custom validators
+
+With custom validators, you can implement business logic in Python. Schema-enforcer will automatically
+load your plugins from the `validator_directory` and run them against your host data.
+
+The validator plugin provides two base classes: ModelValidation and JmesPathModelValidation. The former can be used
+when you want to implement all logic and the latter can be used as a shortcut for jmespath validation.
+
+## BaseValidation
+
+Use this class to implement arbitrary validation logic in Python. In order to work correctly, your Python script must meet
+the following criteria:
+
+1. Exist in the `validator_directory` dir.
+2. Include a subclass of the BaseValidation class to correctly register with schema-enforcer.
+3. Ensure you call `super().__init__()` in your class `__init__` if you override.
+4. Provide a class method in your subclass with the following signature:
+`def validate(data: dict, strict: bool):`
+
+   * Data is a dictionary of variables on a per-host basis.
+   * Strict is set to true when the strict flag is set via the CLI. You can use this to offer strict validation behavior
+   or ignore it if not needed.
+
+The name of your class will be used as the schema-id for mapping purposes. You can override the default schema ID
+by providing a class-level `id` variable.
+
+Helper functions are provided to add pass/fail results:
+
+```
+def add_validation_error(self, message: str, **kwargs):
+    """Add validator error to results.
+    Args:
+      message (str): error message
+      kwargs (optional): additional arguments to add to ValidationResult when required
+    """
+
+def add_validation_pass(self, **kwargs):
+    """Add validator pass to results.
+    Args:
+      kwargs (optional): additional arguments to add to ValidationResult when required
+    """
+```
+In most cases, you will not need to provide kwargs. However, if you find a use case that requires updating other fields
+in the ValidationResult, you can send the key/value pairs to update the result directly. This is for advanced users only.
+
+## JmesPathModelValidation
+
+Use this class for basic validation using [jmespath](https://jmespath.org/) expressions to query specific values in your data. In order to work correctly, your Python script must meet
+the following criteria:
+
+1. Exist in the `validator_directory` dir.
+2. Include a subclass of the JmesPathModelValidation class to correctly register with schema-enforcer.
+3. Provide the following class level variables:
+
+   * `top_level_properties`: Field for mapping of validator to data
+   * `id`: Schema ID to use for reporting purposes (optional - defaults to class name)
+   * `left`: Jmespath expression to query your host data
+   * `right`: Value or a compiled jmespath expression
+   * `operator`: Operator to use for comparison between left and right hand side of expression
+   * `error`: Message to report when validation fails
+
+### Supported operators:
+
+The class provides the following operators for basic use cases:
+
+```
+"gt": int(left) > int(right),
+"gte": int(left) >= int(right),
+"eq": left == right,
+"lt": int(left) < int(right),
+"lte": int(left) <= int(right),
+"contains": right in left,
+```
+
+If you require additional logic or need to compare other types, use the BaseValidation class and create your own validate method.
+
+### Examples:
+
+#### Basic
+```
+from schema_enforcer.schemas.validator import JmesPathModelValidation
+
+class CheckInterface(JmesPathModelValidation):  # pylint: disable=too-few-public-methods
+    top_level_properties = ["interfaces"]
+    id = "CheckInterface"  # pylint: disable=invalid-name
+    left = "interfaces.*[@.type=='core'][] | length([?@])"
+    right = 2
+    operator = "gte"
+    error = "Less than two core interfaces"
+```
+
+#### With compiled jmespath expression
+```
+import jmespath
+from schema_enforcer.schemas.validator import JmesPathModelValidation
+
+
+class CheckInterfaceIPv4(JmesPathModelValidation):  # pylint: disable=too-few-public-methods
+    top_level_properties = ["interfaces"]
+    id = "CheckInterfaceIPv4"  # pylint: disable=invalid-name
+    left = "interfaces.*[@.type=='core'][] | length([?@])"
+    right = jmespath.compile("interfaces.* | length([[email protected]=='core'][].ipv4)")
+    operator = "eq"
+    error = "All core interfaces do not have IPv4 addresses"
+```
+
+## Running validators
+
+Custom validators are run with `schema-enforcer validate` and `schema-enforcer ansible` commands.
+
+You map validators to keys in your data with `top_level_properties` in your subclass or with `schema_enforcer_schema_ids`
+in your data. Schema-enforcer uses the same process to map custom validators and schemas. Refer to the "Mapping Schemas" documentation
+for more details.
+
+### Example - top_level_properties
+
+The CheckInterface validator has a top_level_properties of "interfaces":
+
+```
+class CheckInterface(JmesPathModelValidation):  # pylint: disable=too-few-public-methods
+    top_level_properties = ["interfaces"]
+```
+
+With automapping enabled, this validator will apply to any host with a top-level `interfaces` key in the Ansible host_vars data:
+
+```
+---
+hostname: "az-phx-pe01"
+pair_rtr: "az-phx-pe02"
+interfaces:
+  MgmtEth0/0/CPU0/0:
+    ipv4: "172.16.1.1"
+  Loopback0:
+    ipv4: "192.168.1.1"
+    ipv6: "2001:db8:1::1"
+  GigabitEthernet0/0/0/0:
+    ipv4: "10.1.0.1"
+    ipv6: "2001:db8::"
+    peer: "az-phx-pe02"
+    peer_int: "GigabitEthernet0/0/0/0"
+    type: "core"
+  GigabitEthernet0/0/0/1:
+    ipv4: "10.1.0.37"
+    ipv6: "2001:db8::12"
+    peer: "co-den-p01"
+    peer_int: "GigabitEthernet0/0/0/2"
+    type: "core"
+```
+
+### Example - manual mapping
+
+Alternatively, you can manually map a validator in your Ansible host vars or other data files.
+
+```
+schema_enforcer_automap_default: false
+schema_enforcer_schema_ids:
+  - "CheckInterface"
+```

examples/ansible3/host_vars/az_phx_pe01/base.yml

@@ -0,0 +1,22 @@
+---
+hostname: "az-phx-pe01"
+pair_rtr: "az-phx-pe02"
+upstreams: []
+interfaces:
+  MgmtEth0/0/CPU0/0:
+    ipv4: "172.16.1.1"
+  Loopback0:
+    ipv4: "192.168.1.1"
+    ipv6: "2001:db8:1::1"
+  GigabitEthernet0/0/0/0:
+    ipv4: "10.1.0.1"
+    ipv6: "2001:db8::"
+    peer: "az-phx-pe02"
+    peer_int: "GigabitEthernet0/0/0/0"
+    type: "core"
+  GigabitEthernet0/0/0/1:
+    ipv4: "10.1.0.37"
+    ipv6: "2001:db8::12"
+    peer: "co-den-p01"
+    peer_int: "GigabitEthernet0/0/0/2"
+    type: "core"

examples/ansible3/host_vars/az_phx_pe02/base.yml

@@ -0,0 +1,22 @@
+---
+hostname: "az-phx-pe02"
+pair_rtr: "az-phx-pe01"
+upstreams: []
+interfaces:
+  MgmtEth0/0/CPU0/0:
+    ipv4: "172.16.1.2"
+  Loopback0:
+    ipv4: "192.168.1.2"
+    ipv6: "2001:db8:1::2"
+  GigabitEthernet0/0/0/0:
+    ipv4: "10.1.0.2"
+    ipv6: "2001:db8::1"
+    peer: "az-phx-pe01"
+    peer_int: "GigabitEthernet0/0/0/0"
+    type: "core"
+  GigabitEthernet0/0/0/1:
+    ipv4: "10.1.0.41"
+    ipv6: "2001:db8::14"
+    peer: "co-den-p02"
+    peer_int: "GigabitEthernet0/0/0/2"
+    type: "access"

examples/ansible3/inventory.yml

@@ -0,0 +1,15 @@
+---
+all:
+  vars:
+    ansible_network_os: "iosxr"
+    ansible_user: "cisco"
+    ansible_password: "cisco"
+    ansible_connection: "netconf"
+    ansible_netconf_ssh_config: true
+  children:
+    pe_rtrs:
+      hosts:
+        az_phx_pe01:
+          ansible_host: "172.16.1.1"
+        az_phx_pe02:
+          ansible_host: "172.16.1.2"

examples/ansible3/pyproject.toml

@@ -0,0 +1,2 @@
+[tool.schema_enforcer]
+ansible_inventory = "inventory.yml"

examples/ansible3/validators/check_interfaces.py

@@ -0,0 +1,13 @@
+"""Example validator plugin."""
+from schema_enforcer.schemas.validator import JmesPathModelValidation
+
+
+class CheckInterface(JmesPathModelValidation):  # pylint: disable=too-few-public-methods
+    """Check that each device has more than one core uplink."""
+
+    top_level_properties = ["interfaces"]
+    id = "CheckInterface"  # pylint: disable=invalid-name
+    left = "interfaces.*[@.type=='core'][] | length([?@])"
+    right = 2
+    operator = "gte"
+    error = "Less than two core interfaces"

pyproject.toml

@@ -25,6 +25,7 @@ jsonref = "^0.2"
 pydantic = "^1.6.1"
 rich = "^9.5.1"
 ansible = "^2.8.0"
+jmespath = "^0.10.0"
 
 [tool.poetry.dev-dependencies]
 pytest = "^5.4.1"
@@ -75,6 +76,9 @@ notes = """,
     XXX,
     """
 
+[tool.pylint.SIMILARITIES]
+min-similarity-lines = 15
+
 [tool.pytest.ini_options]
 testpaths = [
     "tests"

schema_enforcer/cli.py

@@ -303,8 +303,9 @@ def ansible(
                 data = hostvars
 
             # Validate host vars against schema
-            for result in schema_obj.validate(data=data, strict=strict):
+            schema_obj.validate(data=data, strict=strict)
 
+            for result in schema_obj.get_results():
                 result.instance_type = "HOST"
                 result.instance_hostname = host.name
 
@@ -314,6 +315,7 @@ def ansible(
 
                 elif result.passed() and show_pass:
                     result.print()
+            schema_obj.clear_results()
 
     if not error_exists:
         print(colored("ALL SCHEMA VALIDATION CHECKS PASSED", "green"))

schema_enforcer/config.py

@@ -27,6 +27,7 @@ class Settings(BaseSettings):  # pylint: disable=too-few-public-methods
     main_directory: str = "schema"
     definition_directory: str = "definitions"
     schema_directory: str = "schemas"
+    validator_directory: str = "validators"
     test_directory: str = "tests"
 
     # Settings specific to the schema files

schema_enforcer/instances/file.py

@@ -128,6 +128,9 @@ def validate(self, schema_manager, strict=False):
         for schema_id, schema in schema_manager.iter_schemas():
             if schema_id not in self.matches:
                 continue
-            errs = itertools.chain(errs, schema.validate(self.get_content(), strict))
+            schema.validate(self.get_content(), strict)
+            results = schema.get_results()
+            errs = itertools.chain(errs, results)
+            schema.clear_results()
 
         return errs