Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(anta): Added test case to verify dynamic vlans source #978

Open
wants to merge 10 commits into
base: main
Choose a base branch
from
1 change: 1 addition & 0 deletions anta/custom_types.py
Original file line number Diff line number Diff line change
Expand Up @@ -238,3 +238,4 @@ def validate_regex(value: str) -> str:
"Route Cache Route",
"CBF Leaked Route",
]
DynamicVLANSource = Literal["dmf", "dot1x", "dynvtep", "evpn", "mlag", "mlagsync", "mvpn", "swfwd", "vccbfd"]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
DynamicVLANSource = Literal["dmf", "dot1x", "dynvtep", "evpn", "mlag", "mlagsync", "mvpn", "swfwd", "vccbfd"]
DynamicVlanSource = Literal["dmf", "dot1x", "dynvtep", "evpn", "mlag", "mlagsync", "mvpn", "swfwd", "vccbfd"]

77 changes: 76 additions & 1 deletion anta/tests/vlan.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,9 @@

from typing import TYPE_CHECKING, ClassVar, Literal

from anta.custom_types import Vlan
from pydantic import ConfigDict

from anta.custom_types import DynamicVLANSource, Vlan
from anta.models import AntaCommand, AntaTest
from anta.tools import get_failed_logs, get_value

Expand Down Expand Up @@ -68,3 +70,76 @@ def test(self) -> None:
self.result.is_failure(failed_log)
else:
self.result.is_success()


class VerifyDynamicVlanSource(AntaTest):
"""Verifies dynamic VLAN(s) source.

This test performs the following checks for specified dynamic VLAN(s):

1. Ensures that dynamic VLAN(s) are properly configured in the system.
2. Confirms that dynamic VLAN(s) are enabled for any/all the designated sources and disabled for all others.
3. When strict mode is enabled (`strict: true`):
- Dynamic VLAN(s) are enabled for all designated sources.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you clarify the intent of the test?

It could be difficult for users to know which services will appear in the output. Also, a service (or source) being present in the dynamic VLANs output just means it's registered as a potential consumer and having no VLANs allocated (empty vlanIds list) doesn't necessarily mean it's an issue. Example: EVPN being "activated" but showing an empty vlanIds list is perfectly normal if it doesn't need any dynamic VLANs.

If the goal is to check that the provided services have dynamic VLANs registered, the test logic could be simplified a lot by looping over the input sources list, check if the source is present, if it is check if its vlanIds list is not empty.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The actual test criteria is If there are any dynamic VLANs, then these should only have source of "evpn" or "mlagsync" ie. all other sources must be "NONE".
We are taking the sources as input and providing an option for exact match. On the other hand, if strict is set to False(default option), the test will pass as long as the dynamic VLANs are assigned to any of the input sources.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok in that case we need to update the docstring and logic to reflect this because, in my opinion, a service with an empty vlanIds list doesn't mean it's not configured/enabled. It just means the service doesn't use any dynamic VLANs.

The test should pass if all provided source(s) have dynamic VLANs registered (vlanIds not empty), if not, we just say there are no dynamic VLANs for the source(s) concerned. If strict is set, the test would fail if other sources (not provided by the user) have dynamic VLANs registered.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Logic:

  1. Strict: False
  • If all provided source(s) have dynamic VLANs registered, Test should pass
  • If not then passing the test as there are no dynamic VLANs for the source(s) concerned.
  • If dynamic vlans are assigned to sources other than provided, Test should fail.
  1. Strict: True
  • If all provided source(s) have dynamic VLANs registered, Test should pass
  • If not, Then test should fail saying dynamic vlans are not assigned to given sources.
  • If dynamic vlans are assigned to sources other than provided, Test should fail.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apologies if I was not super clear. Here's what I have in mind:

  1. Strict: False
  • If all provided source(s) have dynamic VLANs registered, test should pass.
  • If not, then test should fail saying dynamic VLANs are not assigned to given sources.
  • If dynamic VLANs are assigned to sources other than provided, test should pass (we don't care about other sources not provided by the user, we just explicity check the provided ones).
  1. Strict: True
  • If all provided source(s) have dynamic VLANs registered, test should pass.
  • If not, then test should fail saying dynamic VLANs are not assigned to given sources.
  • If dynamic VLANs are assigned to sources other than provided, test should fail (we do care about other sources now, we are not expecting other dynamic VLANs on other sources).

Copy link
Collaborator

@vitthalmagadum vitthalmagadum Jan 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @carl-baillargeon for clarifying the additional context and explanation, makes things much clearer.


Expected Results
vitthalmagadum marked this conversation as resolved.
Show resolved Hide resolved
----------------
* Success: The test will pass if all of the following conditions are met:
- The dynamic VLAN(s) are properly configured in the system.
- The dynamic VLAN(s) are enabled for any/all of the designated sources and disabled for all others.
- In strict mode, dynamic VLAN(s) are enabled for all designated sources.
* Failure: The test will fail if any of the following conditions is met:
- The dynamic VLAN(s) are disabled on all designated sources, or active on non designated sources.
- In strict mode, dynamic VLAN(s) are disabled on any of the designated sources.
* Skipped: The test will Skip if the following conditions is met:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* Skipped: The test will Skip if the following conditions is met:
* Skipped: The test will skip if the following conditions is met:

- Dynamic VLAN(s) are not configured on the device.

Examples
--------
```yaml
anta.tests.vlan:
- VerifyDynamicVlanSource:
source:
- evpn
- mlagsync
strict: False
```
"""

categories: ClassVar[list[str]] = ["vlan"]
commands: ClassVar[list[AntaCommand | AntaTemplate]] = [AntaCommand(command="show vlan dynamic", revision=1)]

class Input(AntaTest.Input):
"""Input model for the VerifyDynamicVlanSource test."""

model_config = ConfigDict(extra="forbid")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
model_config = ConfigDict(extra="forbid")

Not needed here since it is already in the parent class AntaTest.Input.

source: list[DynamicVLANSource]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
source: list[DynamicVLANSource]
sources: list[DynamicVLANSource]

"""The dynamic VLAN(s) source list."""
strict: bool = False
"""If True, requires exact match of the provided dynamic VLAN(s) sources, Defaults to `False`"""
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
"""If True, requires exact match of the provided dynamic VLAN(s) sources, Defaults to `False`"""
"""If True, requires exact match of the provided dynamic VLAN(s) sources. Defaults to `False`."""


@AntaTest.anta_test
def test(self) -> None:
"""Main test function for VerifyDynamicVlanSource."""
self.result.is_success()
command_output = self.instance_commands[0].json_output
dynamic_vlans = command_output.get("dynamicVlans", {})

actual_source = [source for source, data in dynamic_vlans.items() if data.get("vlanIds")]
# If the dynamic vlans are not configured, skipping the test.
if not actual_source:
self.result.is_skipped("Dynamic VLANs are not configured")
return

expected_source = self.inputs.source
str_expected_source = ", ".join(expected_source)
str_actual_source = ", ".join(actual_source)

# If strict flag is True and dynamic VLAN(s) are disabled on any of the designated sources, test fails.
if self.inputs.strict and sorted(actual_source) != (expected_source):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could lead to false negatives if expected_source is unsorted.

self.result.is_failure(f"Dynamic VLAN(s) source mismatch - Expected: {str_expected_source} Actual: {str_actual_source}")
return

# The dynamic VLAN(s) are disabled on all designated sources, or active on non designated sources, test fails.
if not set(actual_source).issubset(expected_source):
vitthalmagadum marked this conversation as resolved.
Show resolved Hide resolved
self.result.is_failure(f"Dynamic VLAN(s) source mismatch - {str_actual_source} are not in the expected sources: {str_expected_source}.")
6 changes: 6 additions & 0 deletions examples/tests.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -836,6 +836,12 @@ anta.tests.system:
# Verifies the device uptime.
minimum: 86400
anta.tests.vlan:
- VerifyDynamicVlanSource:
# Verifies dynamic VLAN(s) source.
source:
- evpn
- mlagsync
strict: False
- VerifyVlanInternalPolicy:
# Verifies the VLAN internal allocation policy and the range of VLANs.
policy: ascending
Expand Down
51 changes: 50 additions & 1 deletion tests/units/anta_tests/test_vlan.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@

from typing import Any

from anta.tests.vlan import VerifyVlanInternalPolicy
from anta.tests.vlan import VerifyDynamicVlanSource, VerifyVlanInternalPolicy
from tests.units.anta_tests import test

DATA: list[dict[str, Any]] = [
Expand All @@ -33,4 +33,53 @@
],
},
},
{
"name": "success",
"test": VerifyDynamicVlanSource,
"eos_data": [{"dynamicVlans": {"evpn": {"vlanIds": [1199]}, "mlagsync": {"vlanIds": []}}}],
"inputs": {"source": ["evpn", "mlagsync"], "strict": False},
"expected": {"result": "success"},
},
{
"name": "failure-no-dynamic-vlans",
"test": VerifyDynamicVlanSource,
"eos_data": [{"dynamicVlans": {}}],
"inputs": {"source": ["evpn", "mlagsync"], "strict": False},
"expected": {"result": "skipped", "messages": ["Dynamic VLANs are not configured"]},
},
{
"name": "failure-dynamic-vlan-source-invalid",
"test": VerifyDynamicVlanSource,
"eos_data": [{"dynamicVlans": {"vccbfd": {"vlanIds": [1500]}, "mlagsync": {"vlanIds": [1501]}}}],
"inputs": {"source": ["evpn", "mlagsync"], "strict": False},
"expected": {"result": "failure", "messages": ["Dynamic VLAN(s) source mismatch - vccbfd, mlagsync are not in the expected sources: evpn, mlagsync."]},
},
{
"name": "failure-any-source-match-additional-source-found",
"test": VerifyDynamicVlanSource,
"eos_data": [{"dynamicVlans": {"evpn": {"vlanIds": [1199]}, "mlagsync": {"vlanIds": [1501]}, "vccbfd": {"vlanIds": [1500]}}}],
"inputs": {"source": ["evpn", "mlagsync"], "strict": False},
"expected": {"result": "failure", "messages": ["Dynamic VLAN(s) source mismatch - evpn, mlagsync, vccbfd are not in the expected sources: evpn, mlagsync."]},
},
{
"name": "success-strict-mode",
"test": VerifyDynamicVlanSource,
"eos_data": [{"dynamicVlans": {"evpn": {"vlanIds": [1199]}, "mlagsync": {"vlanIds": [1502]}}}],
"inputs": {"source": ["evpn", "mlagsync"], "strict": True},
"expected": {"result": "success"},
},
{
"name": "failure-all-source-exact-match-additional-source-found",
"test": VerifyDynamicVlanSource,
"eos_data": [{"dynamicVlans": {"evpn": {"vlanIds": [1199]}, "mlagsync": {"vlanIds": [1500]}, "vccbfd": {"vlanIds": [1500]}}}],
"inputs": {"source": ["evpn", "mlagsync"], "strict": True},
"expected": {"result": "failure", "messages": ["Dynamic VLAN(s) source mismatch - Expected: evpn, mlagsync Actual: evpn, mlagsync, vccbfd"]},
},
{
"name": "failure-all-source-exact-match-expected-source-not-found",
"test": VerifyDynamicVlanSource,
"eos_data": [{"dynamicVlans": {"evpn": {"vlanIds": [1199]}, "mlagsync": {"vlanIds": []}}}],
"inputs": {"source": ["evpn", "mlagsync"], "strict": True},
"expected": {"result": "failure", "messages": ["Dynamic VLAN(s) source mismatch - Expected: evpn, mlagsync Actual: evpn"]},
},
]
Loading