Merge pull request #100 from networktocode/develop-2.0

Release 2.0

Author: dgarros

README.md

@@ -26,8 +26,10 @@ pip install ntc-netbox-plugin-onboarding
 systemctl restart netbox netbox-rq
 ```
 
-> The plugin is compatible with NetBox 2.8.1 and higher
- 
+> The ntc-netbox-plugin-onboarding v1.3 is compatible with NetBox 2.8
+
+> The ntc-netbox-plugin-onboarding v2 is compatible with NetBox 2.8 and NetBox 2.9
+
 To ensure NetBox Onboarding plugin is automatically re-installed during future upgrades, create a file named `local_requirements.txt` (if not already existing) in the NetBox root directory (alongside `requirements.txt`) and list the `ntc-netbox-plugin-onboarding` package:
 
 ```no-highlight
@@ -64,26 +66,38 @@ The plugin behavior can be controlled with the following list of settings
 - `default_device_role_color` string (default FF0000), color assigned to the device role if it needs to be created.
 - `default_management_interface` string (default "PLACEHOLDER"), name of the management interface that will be created, if one can't be identified on the device.
 - `default_management_prefix_length` integer ( default 0), length of the prefix that will be used for the management IP address, if the IP can't be found.
+- `skip_device_type_on_update` boolean (default False), If True, an existing NetBox device will not get its device type updated. If False, device type will be updated with one discovered on a device.
+- `skip_manufacturer_on_update` boolean (default False), If True, an existing NetBox device will not get its manufacturer updated. If False, manufacturer will be updated with one discovered on a device.
 - `platform_map` (dictionary), mapping of an **auto-detected** Netmiko platform to the **NetBox slug** name of your Platform. The dictionary should be in the format:
     ```python
     {
-      <Netmiko Platform>: <NetBox Slug> 
+      <Netmiko Platform>: <NetBox Slug>
+    }
+    ```
+- `onboarding_extensions_map` (dictionary), mapping of a NAPALM driver name to the loadable Python module used as an onboarding extension. The dictionary should be in the format:
+    ```python
+    {
+      <Napalm Driver Name>: <Loadable Python Module>
     }
     ```
+- `object_match_strategy` (string), defines the method for searching models. There are
+currently two strategies, strict and loose. Strict has to be a direct match, normally 
+using a slug. Loose allows a range of search criteria to match a single object. If multiple
+objects are returned an error is raised. 
 
 ## Usage
 
 ### Preparation
 
-To work properly the plugin needs to know the Site, Platform, Device Type, Device Role of each
-device as well as its primary IP address or DNS Name. It's recommended to create these objects in
-NetBox ahead of time and to provide them when you want to start the onboarding process.
+To properly onboard a device, the plugin needs to only know the Site as well as device's primary IP address or DNS Name.
 
 > For DNS Name Resolution to work, the instance of NetBox must be able to resolve the name of the
 > device to IP address.
 
+Providing other attributes (`Platform`, `Device Type`, `Device Role`) is optional - if any of these attributes is provided, plugin will use provided value for the onboarded device.
 If `Platform`, `Device Type` and/or `Device Role` are not provided, the plugin will try to identify these information automatically and, based on the settings, it can create them in NetBox as needed.
-> If the Platform is provided, it must contains a valid Napalm driver available to the worker in Python
+> If the Platform is provided, it must point to an existing NetBox Platform. NAPALM driver of this platform will be used only if it is defined for the platform in NetBox.
+> To use a preferred NAPALM driver, either define it in NetBox per platform or in the plugins settings under `platform_map`
 
 ### Onboard a new device
 

development/base_configuration.py

@@ -114,7 +114,20 @@
 
 # Enable custom logging. Please see the Django documentation for detailed guidance on configuring custom logs:
 #   https://docs.djangoproject.com/en/1.11/topics/logging/
-LOGGING = {}
+LOGGING = {
+    "version": 1,
+    "disable_existing_loggers": False,
+    "formatters": {"rq_console": {"format": "%(asctime)s %(message)s", "datefmt": "%H:%M:%S",},},
+    "handlers": {
+        "rq_console": {
+            "level": "DEBUG",
+            "class": "rq.utils.ColorizingStreamHandler",
+            "formatter": "rq_console",
+            "exclude": ["%(asctime)s"],
+        },
+    },
+    "loggers": {"rq.worker": {"handlers": ["rq_console"], "level": "DEBUG"},},
+}
 
 # Setting this to True will permit only authenticated users to access any part of NetBox. By default, anonymous users
 # are permitted to access most data in NetBox (excluding secrets) but not make any changes.

docs/examples/example_ios_set_device_role.py

@@ -0,0 +1,96 @@
+"""Example of custom onboarding class.
+
+(c) 2020 Network To Code
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+  http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+"""
+
+from netbox_onboarding.netbox_keeper import NetboxKeeper
+from netbox_onboarding.onboarding.onboarding import Onboarding
+
+
+class MyOnboardingClass(Onboarding):
+    """Custom onboarding class example.
+
+    Main purpose of this class is to access and modify the onboarding_kwargs.
+    By accessing the onboarding kwargs, user gains ability to modify
+    onboarding parameters before the objects are created in NetBox.
+
+    This class adds the get_device_role method that does the static
+     string comparison and returns the device role.
+    """
+
+    def run(self, onboarding_kwargs):
+        """Ensures network device."""
+        # Access hostname from onboarding_kwargs and get device role automatically
+        device_new_role = self.get_device_role(hostname=onboarding_kwargs["netdev_hostname"])
+
+        # Update the device role in onboarding kwargs dictionary
+        onboarding_kwargs["netdev_nb_role_slug"] = device_new_role
+
+        nb_k = NetboxKeeper(**onboarding_kwargs)
+        nb_k.ensure_device()
+
+        self.created_device = nb_k.device
+
+    @staticmethod
+    def get_device_role(hostname):
+        """Returns the device role based on hostname data.
+
+        This is a static analysis of hostname string content only
+        """
+        hostname_lower = hostname.lower()
+        if ("rtr" in hostname_lower) or ("router" in hostname_lower):
+            role = "router"
+        elif ("sw" in hostname_lower) or ("switch" in hostname_lower):
+            role = "switch"
+        elif ("fw" in hostname_lower) or ("firewall" in hostname_lower):
+            role = "firewall"
+        elif "dc" in hostname_lower:
+            role = "datacenter"
+        else:
+            role = "generic"
+
+        return role
+
+
+class OnboardingDriverExtensions:
+    """This is an example of a custom onboarding driver extension.
+
+    This extension sets the onboarding_class to MyOnboardingClass,
+     which is an example class of how to access and modify the device
+     role automatically through the onboarding process.
+    """
+
+    def __init__(self, napalm_device):
+        """Inits the class."""
+        self.napalm_device = napalm_device
+        self.onboarding_class = MyOnboardingClass
+        self.ext_result = None
+
+    def get_onboarding_class(self):
+        """Return onboarding class for IOS driver.
+
+        Currently supported is Standalone Onboarding Process
+
+        Result of this method is used by the OnboardingManager to
+        initiate the instance of the onboarding class.
+        """
+        return self.onboarding_class
+
+    def get_ext_result(self):
+        """This method is used to store any object as a return value.
+
+        Result of this method is passed to the onboarding class as
+        driver_addon_result argument.
+
+        :return: Any()
+        """
+        return self.ext_result

docs/release-notes/version-2.0.md

@@ -0,0 +1,24 @@
+# ntc-netbox-plugin-onboarding v2.0 Release Notes
+
+## v2.0
+
+### Enhancements
+
+* NetBox 2.9 support - Supported releases 2.8 and 2.9
+* Onboarding extensions - Customizable onboarding process through Python modules.
+* Onboarding details exposed in a device view - Date, Status, Last success and Latest task id related to the onboarded device are presented under the device view.
+* Onboarding task view - Onboarding details exposed in a dedicated view, including NetBox's ChangeLog.
+* Onboarding Changelog - Onboarding uses NetBox's ChangeLog to display user and changes made to the Onboarding Task object.
+* Skip onboarding feature - New attribute in the OnboardingDevice model allows to skip the onboarding request on devices with disabled onboarding setting.
+
+### Bug Fixes
+
+* Fixed race condition in `worker.py`
+* Improved logging
+
+### Additional Changes
+
+* Platform map now includes NAPALM drivers as defined in NetBox
+* Tests have been refactored to inherit NetBox's tests
+* Onboarding process will update the Device found by the IP-address lookup. In case of no existing device with onboarded IP-address is found in NetBox, onboarding might update the existing NetBox' looking up by network device's hostname.
+* Onboarding will raise Exception when `create_device_type_if_missing` is set to `False` for existing Device with DeviceType mismatch (behaviour pre https://github.com/networktocode/ntc-netbox-plugin-onboarding/issues/74)

netbox_onboarding/__init__.py

@@ -12,7 +12,7 @@
 limitations under the License.
 """
 
-__version__ = "1.3.0"
+__version__ = "2.0.0"
 
 from extras.plugins import PluginConfig
 
@@ -39,7 +39,11 @@ class OnboardingConfig(PluginConfig):
         "default_management_prefix_length": 0,
         "default_device_status": "active",
         "create_management_interface_if_missing": True,
+        "skip_device_type_on_update": False,
+        "skip_manufacturer_on_update": False,
         "platform_map": {},
+        "onboarding_extensions_map": {"ios": "netbox_onboarding.onboarding_extensions.ios",},
+        "object_match_strategy": "loose",
     }
     caching_config = {}
 

netbox_onboarding/admin.py

@@ -32,5 +32,5 @@ class OnboardingTaskAdmin(admin.ModelAdmin):
         "failed_reason",
         "port",
         "timeout",
-        "created_on",
+        "created",
     )

netbox_onboarding/choices.py

@@ -22,12 +22,14 @@ class OnboardingStatusChoices(ChoiceSet):
     STATUS_PENDING = "pending"
     STATUS_RUNNING = "running"
     STATUS_SUCCEEDED = "succeeded"
+    STATUS_SKIPPED = "skipped"
 
     CHOICES = (
         (STATUS_FAILED, "failed"),
         (STATUS_PENDING, "pending"),
         (STATUS_RUNNING, "running"),
         (STATUS_SUCCEEDED, "succeeded"),
+        (STATUS_SKIPPED, "skipped"),
     )
 
 

netbox_onboarding/constants.py

@@ -1,5 +1,6 @@
 """Constants for netbox_onboarding plugin."""
-NETMIKO_TO_NAPALM = {
+
+NETMIKO_TO_NAPALM_STATIC = {
     "cisco_ios": "ios",
     "cisco_nxos": "nxos_ssh",
     "arista_eos": "eos",

netbox_onboarding/exceptions.py

@@ -0,0 +1,39 @@
+"""Exceptions.
+
+(c) 2020 Network To Code
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+  http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+"""
+
+
+class OnboardException(Exception):
+    """A failure occurred during the onboarding process.
+
+    The exception includes a reason "slug" as defined below as well as a humanized message.
+    """
+
+    REASONS = (
+        "fail-config",  # config provided is not valid
+        "fail-connect",  # device is unreachable at IP:PORT
+        "fail-execute",  # unable to execute device/API command
+        "fail-login",  # bad username/password
+        "fail-dns",  # failed to get IP address from name resolution
+        "fail-general",  # other error
+    )
+
+    def __init__(self, reason, message, **kwargs):
+        """Exception Init."""
+        super(OnboardException, self).__init__(kwargs)
+        self.reason = reason
+        self.message = message
+
+    def __str__(self):
+        """Exception __str__."""
+        return f"{self.__class__.__name__}: {self.reason}: {self.message}"

netbox_onboarding/filters.py

@@ -26,8 +26,6 @@ class OnboardingTaskFilter(NameSlugSearchFilterSet):
 
     q = django_filters.CharFilter(method="search", label="Search",)
 
-    site_id = django_filters.ModelMultipleChoiceFilter(queryset=Site.objects.all(), label="Site (ID)",)
-
     site = django_filters.ModelMultipleChoiceFilter(
         field_name="site__slug", queryset=Site.objects.all(), to_field_name="slug", label="Site (slug)",
     )
@@ -46,7 +44,7 @@ class Meta:  # noqa: D106 "Missing docstring in public nested class"
         model = OnboardingTask
         fields = ["id", "site", "site_id", "platform", "role", "status", "failed_reason"]
 
-    def search(self, queryset, name, value):
+    def search(self, queryset, name, value):  # pylint: disable=unused-argument, no-self-use
         """Perform the filtered search."""
         if not value.strip():
             return queryset
@@ -55,7 +53,7 @@ def search(self, queryset, name, value):
             | Q(ip_address__icontains=value)
             | Q(site__name__icontains=value)
             | Q(platform__name__icontains=value)
-            | Q(device__icontains=value)
+            | Q(created_device__name__icontains=value)
             | Q(status__icontains=value)
             | Q(failed_reason__icontains=value)
             | Q(message__icontains=value)

netbox_onboarding/forms.py

@@ -13,11 +13,11 @@
 """
 
 from django import forms
+from django.db import transaction
 from django_rq import get_queue
 
-from utilities.forms import BootstrapMixin
+from utilities.forms import BootstrapMixin, CSVModelForm
 from dcim.models import Site, Platform, DeviceRole, DeviceType
-from extras.forms import CustomFieldModelCSVForm
 
 from .models import OnboardingTask
 from .choices import OnboardingStatusChoices, OnboardingFailChoices
@@ -33,7 +33,7 @@ class OnboardingTaskForm(BootstrapMixin, forms.ModelForm):
         required=True, label="IP address", help_text="IP Address/DNS Name of the device to onboard"
     )
 
-    site = forms.ModelChoiceField(required=True, queryset=Site.objects.all(), to_field_name="slug")
+    site = forms.ModelChoiceField(required=True, queryset=Site.objects.all())
 
     username = forms.CharField(required=False, help_text="Device username (will not be stored in database)")
     password = forms.CharField(
@@ -106,7 +106,7 @@ class Meta:  # noqa: D106 "Missing docstring in public nested class"
         fields = ["q", "site", "platform", "status", "failed_reason"]
 
 
-class OnboardingTaskFeedCSVForm(CustomFieldModelCSVForm):
+class OnboardingTaskFeedCSVForm(CSVModelForm):
     """Form for entering CSV to bulk-import OnboardingTask entries."""
 
     site = forms.ModelChoiceField(
@@ -149,12 +149,21 @@ class OnboardingTaskFeedCSVForm(CustomFieldModelCSVForm):
 
     class Meta:  # noqa: D106 "Missing docstring in public nested class"
         model = OnboardingTask
-        fields = OnboardingTask.csv_headers
+        fields = [
+            "site",
+            "ip_address",
+            "port",
+            "timeout",
+            "platform",
+            "role",
+        ]
 
     def save(self, commit=True, **kwargs):
         """Save the model, and add it and the associated credentials to the onboarding worker queue."""
         model = super().save(commit=commit, **kwargs)
         if commit:
             credentials = Credentials(self.data.get("username"), self.data.get("password"), self.data.get("secret"))
-            get_queue("default").enqueue("netbox_onboarding.worker.onboard_device", model.pk, credentials)
+            transaction.on_commit(
+                lambda: get_queue("default").enqueue("netbox_onboarding.worker.onboard_device", model.pk, credentials)
+            )
         return model

netbox_onboarding/metrics.py

@@ -0,0 +1,18 @@
+"""Plugin additions to the NetBox navigation menu.
+
+(c) 2020 Network To Code
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+  http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+"""
+from prometheus_client import Counter
+
+onboardingtask_results_counter = Counter(
+    name="onboardingtask_results_total", documentation="Count of results for Onboarding Task", labelnames=("status",)
+)